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FOREWORD 


This  report  documents  the  methodology  and  lessons  learned  in  the  development  of  the 
Innovative  Tools  and  Techniques  for  Brigade  and  Below  Staff  Training  -  Battle  Staff  Training 
System  (ITTBBST-BSTS).  The  ITTBBST-BSTS  was  developed  as  part  of  the  US.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences’  ongoing  research  for  the  Force  XXI 
Training  Program  to  establish  innovative  methodologies  for  training  combined  arms  forces.  It 
was  designed  to  address  an  identified  weakness  in  brigade  and  battalion  battle  staff 
S5mchronization  of  battlefield  operating  systems. 

The  ITTBBST-BSTS  trains  selected  staff  officers  and  commanders  in  current  Army 
battlefield  doctrine,  allows  them  to  practice  using  that  doctrine  within  a  combat  scenario 
environment,  and  records  their  performance  for  analysis  and  improvement.  The  28  training 
support  packages  (TSPs)  in  ITTBBST-BSTS  enable  officers  to  gain  a  good  basic  grounding  in 
their  own  staff  position  and  cross  train  in  others  as  well.  This  will  better  prepare  the  officers  to 
perform  as  effective  combat  team  members. 


ZITA  M.  SIMUTIS  EDGAR  M.  JOHNSON 

Technical  Director  Director 
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BATTLE  STAFF  TRAINING  SYSTEM  IN  SUPPORT  OF  FORCE  XXI 
TRAINING  PROGRAM:  METHODOLGY  AND  LESSONS  LEARNED 

EXEUTIVE  SUMMARY 


Research  Requirement; 

The  Army  needs  structured  training  support  packages  (TSPs)  to  support  training  and 
assessment  of  individual  commanders  and  staffs  at  armored  and  mechanized  infantry  battalion  and 
brigade  headquarters.  Evaluation  of  training  reveals  that  most  brigade  and  battalion  staffs  exhibit 
shortcomings  in  basic  staff  skills  required  to  prepare  and  conduct  successful  operations  at  the 
National  Training  Center  (NTC).  Although  some  criticize  brigade  and  battalion  commanders  for 
making  excuses  for  poor  performance  at  the  NTC,  many  hindrances  to  training  —  such  as  a  lack  of 
training  tools  or  resources  —  do  detract  from  the  sort  of  training  that  would  develop  an  effective 
warfighting  team  from  a  brigade  or  battalion  staff 

Procedure: 

Innovative  Tools  and  Techniques  for  Brigade  and  Below  Staff  Training  -  Battle  Staff 
Training  System  (ITTBBST-BSTS)  was  developed  through  the  use  of  the  Systems  Approach  to 
Training  (SAT)  process  as  applied  to  the  design  and  development  of  multimedia  computer-based 
instruction  (CBI)  and  text  lessons.  The  SAT  cycle  is  a  five-step  loop  of  analysis,  design, 
development,  implementation,  and  evaluation. 

Each  job  was  analyzed  in  light  of  the  tasks  required  of  the  incumbent  in  the  performance 
of  warfighting  duties.  The  jobs  analyzed  were  the  key  battalion  and  brigade  battle  staff  positions, 
including  commanders.  To  these  26  positions  was  added  a  common  core  of  critical  warfighting 
skills  required  by  all  commanders  and  the  members  of  their  respective  staffs.  The  analysis 
included  a  review  of  previous  BSTS  job  analyses,  review  of  applicable  doctrinal  publications, 
review  of  lessons  learned  available  from  published  material  or  military  subject  matter  experts 
(SMEs),  and  review  and  approval  by  the  customer. 

The  design  process  used  the  results  of  the  job  analyses  to  determine  tasks,  identify  which 
ones  were  critical,  produce  instructional  objectives,  develop  criteria  test  items,  prepare  course 
maps,  develop  course  strategies,  and  prepare  examinations.  Formative  evaluation  was  initiated 
early  in  the  design  phase  and  continued  throughout  development.  Development  of  course 
material  was  based  on  the  analysis  and  design  completed  by  the  research  team.  Prototypical  TSPs 
contained,  not  only  instructional  material,  but  a  comprehensive  assessment  component  (COMPS) 
and  remediation  training  (when  evaluated  as  necessary).  ITTBBST-BSTS  also  included  a  User’s 
Guide  and  a  System  Administrator’s  Guide.  The  prototype  courses  were  implemented  on  a  trial 
basis  through  SME  and  target  audience  testing  at  Fort  Knox,  KY.  Results  of  testing  were  used  to 
revise  each  TSP. 


Findings; 


Based  on  evaluation  of  target  audience  tester  comments,  staff  officers  are  receptive  to  new 
training  techniques.  Based  on  tester  ratings,  CBI  and  text  lessons  were  preferred  over  reading 
doctrinal  references.  Target  audience  representatives  showed  a  marked  improvement  in  their  job 
knowledge  and  how  to  synchronize  battlefield  operating  systems  after  completion  of  the 
appropriate  TSP(s)  as  evidenced  by  test  score  improvement  between  pre  and  post  testing. 
ITTBBST-BSTS  appears  to  be  a  valid  tool  for  addressing  training  weaknesses  within  the 
battalion  and  brigade  staff  of  an  armored  or  mechanized  infantry  unit. 

Utilization  of  Findings: 

This  report  describes  the  development  of  ITTBBST-BSTS  and  the  lessons  learned  during 
the  process.  ITTBBST-BSTS  has  shown  itself  to  be  effective  in  improving  staff  officer  skills  and 
knowledge  in  individual  tasks  as  well  as  battlefield  synchronization  principles.  The  CBI  method 
of  instruction  received  high  value  ratings  from  target  audience  testers  and  would  appear  to  lend 
itself  to  further  development  and  expansion  as  a  method  of  individual  officer  training. 
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BATTLE  STAFF  TRAINING  SYSTEM  IN  SUPPORT  OF 
FORCE  XXI  TRAINING  PROGRAM: 
METHODOLOGY  AND  LESSONS  LEARNED 


INTRODUCTION 

The  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  has 
conducted  research  supporting  the  development  of  training  strategies  for  brigade  and  battalion 
level  combined  arms  combat  teams,  crews  or  staff  groups,  and  individuals.  This  is  a  major  focus 
of  the  Force  XXI  Training  Program  (FXXITP),  a  program  designed  to  carry  the  Army's  training 
capabilities  into  the  21st  century  (U.S.  Army  Training  and  Doctrine  Command  (TRADOC), 


In  support  of  FXXITP,  ARI  initiated  the  Innovative  Tools  and  Techniques  for  Brigade 
and  Below  StaffTraining  (ITTBBST)  program.  Using  a  doctrinally-based  performance 
taxonomy,  a  key  ITTBBST  thrust  was  analyzinjg  Battlefield  Functions  ^Fs)  to  establish  a 
common  basis  for  structuring  individual  and  unit  training  programs.  At  the  next  level,  paper- 
based  and  multimedia  computer-based  instruction  (CBI)  in  the  form  of  Battle  StaffTraining 
System  (BSTS)  courses  were  developed  to  train  individual  staff  members  on  skills  required  for 
various  staff  functions.  Capitalizing  on  advanced  workstation  technology,  the  last  level  of  this 
program  involved  development  of  an  innovative  tool  for  training  small  groups  of  staff  officers  in 
the  Staff  Group  Trainer  (SGT).  The  SGT  serves  as  a  bridge  between  individual  staff  training 
provided  by  the  BSTS  and  training  exercises  designed  for  the  integrated  staff. 

Prior  to  collective  training,  staff  officers  must  master  individual  staff  skills  and  their  role  in 
synchronization  of  battlefield  operating  systems.  The  tool  developed  for  this  training  is  BSTS. 
This  report  provides  methodology  and  lessons  learned  in  preparing  BSTS  courses  for  use  in  the 
FXXITP  through  the  ITTBBST-BSTS  project. 

The  ITTBBST-BSTS  is  a  combination  of  text  and  CBI  composed  of  two  sets  of 
fonctional  area  training  packages  for  battalion  and  brigade  level  staff  officers.  The  packages 
include  computer-based  materials,  text-based  instruction  and  a  training  management  system.  The 
ITTBBST-BSTS  was  developed  for  use  by  the  total  force.  Active  and  Reserve,  and  designed  for 
use  in  a  local  area  network,  wide  area  network,  or  stand  alone  computer  mode.  The  work  was 
performed  by  BDM  Federal,  Inc.,  fi'om  January  1996  through  May  1997  under  contract  to  the 
ARI  Armored  Forces  Research  Unit. 


Background 

The  U.S.  Army  currently  faces  the  challenge  of  meeting  complex,  highly  diverse  training 
requirements  with  increasingly  constrained  resources  (Brown,  1994).  Resource  constraints  result 
from  increased  costs  of  acquiring  and  operating  advanced  technology  weapon  systems, 
accompanied  by  force  downsizing  and  shrinking  defense  budgets.  To  meet  the  challenge  of  doing 
more  with  less.  Army  trainers  have  begun  to  develop  innovative  techniques  for  assisting  units  in 
achieving  their  training  goals. 

Training  research  conducted  by  ARI  forms  the  foundation  upon  which  the  FXXITP  is 
being  built,  addressing  maneuver  element  and  staff  training  at  the  individual,  crew/group,  and  unit 
levels.  Much  of  the  ARI  work  centers  around  simulation-based  training  programs  established  for 
the  Reserve  and  Active  Components  (e.g.,  C.  H.  Campbell,  R.  C.  Campbell,  Sanders,  Flynn,  & 
Myers,  1995),  accomplished  by  the  Armored  Forces  Research  Unit  at  Fort  Knox.  Work  on  battle 
staff  training  using  these  programs  was  conducted  at  the  Infantry  Forces  Research  Unit  at  Fort 
Benning.  In  parallel,  the  Unit  Collective  Training  Research  Unit  at  the  Presidio  of  Monterey 
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performed  a  large  body  of  research  on  BFs  (e.g.,  Harrison,  1995),  formerly  known  as  Critical 
Combat  Functions  (CCFs). 

The  Force  XXI  Training  Development  (TD)  Vision  (U.S.  Army  Training  and  Doctrine 
Command,  1995)  states  the  Army’s  TD  process  is  the  Systems  Approach  to  Training  (SAT), 
conducted  at  training/task  proponent  schools  that  results  in  collective,  individual  (WARRIOR), 
and  Army  modernization  training  products.  “The  mission  of  WARRIOR  XXI  program  is  to 
provide  a  future  architecture  for  the  development  of  training  products  and  policies  to  support  the 
total  spectrum  of  training.”  (U.S.  Army  Training  and  Doctrine  Command,  1996).  Within  the 
WARMOR  XXI  trainin^D  vision,  B  STS  is  a  key  development  program  to  implement  the 
following  subcomponents: 

1 .  Distance  Learning;  A  concept  for  the  delivery  of  training  to  the  soldier,  when  and 
where  needed.  Use  of  extensive  worldwide  corporate  and  government  electronic  networks 
provides  a  range  of  capabilities  for  distributing  learning  in  either  a  synchronous/real  time  or 
asynchronous  mode  from  simple  text  transmissions  to  video  telecordTerencing.  It  allows  the 
soldier  to  use  a  modem  and  standard  conununications  software  to  dial  into  the  Army  Training 
Support  Center  (ATSC)  Bulletin  Board  Service  (BBS)  from  any  location,  24  hours  a  day. 

2.  Diagnostics:  An  automation  tool  which  allows  both  soldiers  and  trainers  to  objectively 
evaluate  current  and  historical  performance  against  standardized  criteria,  using  stated  objectives. 

Evaluation  of  brigades  and  battalions  at  the  National  Training  Center  (NTC)  reveals  that 
brigade  commanders,  battalion  conunanders,  and  staff  officers  at  both  levels  exhibit  serious 
deficiencies  in  synchronizing  the  battle  (Rosenberger,  1995).  Coordination  and  integration  of  all 
the  elements  of  combat  power  available  to  battalion  and  brigade  commanders  has  always  been 
crucial  to  winning  the  battle.  Technology  has  made  weapons  more  accurate  and  lethal  than  a 
decade  ago;  requiring,  not  only  coordination  and  integration,  but  more  precise  teamwork  among 
the  combined  arms  elements.  Synchronization,  the  weak  spot,  includes  coordination  and 
integration,  and  brings  the  fourth  dimension  of  the  battlefield  —  time  --  sharply  into  focus.  Never 
before  has  teamwork  been  so  crucial  to  success.  Conunanders  and  staff  officers  must  fight  the 
battle  as  an  active,  synchronized  whole. 

The  ITTBBST-BSTS  addresses  staff  deficiencies  at  the  individual  task  level.  It  provides  a 
means  for  brigade  and  battalion  commanders  and  their  staffs  to  gain  essential  knowledge,  enabling 
them  to  perform  their  critical  conunand,  control,  communications,  and  synchronization  tasks. 
Successfiil  performance  of  these  tasks  lets  them  feel  the  battlefield  and  improve  their  preparedness 
for  participation  in  collective  training. 

Maneuver  Training  Packages 

ARTs  development  of  implementation  packages  to  support  trmning  of  maneuver  elements 
centered  on  the  use  of  SIMulation  NETworking  (SIMNET),  a  form  of  virtual  simulation.  Using 
SIMNET  capabilities,  the  Simulation-based  Multiechelon  Training  Program  for  Armor  Units 
(SIMUTA)  developed  structured  training  exercises  for  a  variety  of  units.  These  included  armor 
battalions/task  forces  (TFs),  armor  companies/teams,  armor  platoons,  cavalry  troops,  mechanized 
infantry  platoons,  and  scout  platoons  (Campbell  et  al.,  1995).  Scenarios  and  training  support 
packages  (TSPs)  for  two  missions— movement  to  contact  (MTC)  and  defend  in  sector  (DIS)~ 
were  developed  initially  for  execution  by  Reserve  Components.  Follow-on  research  extended  the 
training  packages  to  a  deliberate  attack  (DATK)  mission.  The  ground-breaking  SIMUTA  work 
established  a  library  of  TSPs  organized  in  an  efficient  "turn-key"  system  which  makes  the  most  of 
the  training  schedules  available  to  Reserve  units.  The  ARI  team  extended  the  training  library  to 
the  Active  units  in  the  Combined  Arms  Operations  at  Brigade  Level,  Realistically  Achieved 
through  Simulation  (COBRAS)  program  (The  COBRAS  Team,  1995).  The  methodology  and 
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lessons  learned  from  the  SIMUTA  and  COBRAS  efforts  (Hoffman,  Graves,  Roger,  Flynn,  & 
Sever,  1995)  provide  a  foundation  for  future  expansion  of  simulation-based  maneuver  training, 
including  valuable  techniques  for  integrating  exercises  across  echelons.  As  a  lead-in  for  these 
collective  TSPs,  ITTBBST-BSTS  uses  the  same  operational  scenarios  to  drive  practical  exercises, 
to  identify  synchronization  requirements  and  as  the  foundation  for  the  end  of  course 
comprehensive  exams.  This  commonality  is  intended  to  facilitate  the  transition  from  individual  to 
collective  training. 


Individual  Staff  Training  Packages 

For  individual  staff  training,  the  BSTS  at  brigade  (BDE-BSTS)  (Andre  &  Salter,  1996a) 
and  battalion  (BN-BSTS)  (Andre  &  Salter,  1995)  uses  text-based  and  multimedia  CBI  methods  to 
train  individual  staff  members  on  skills  required  for  various  staff  functions.  A  library  of  courses 
covers  a  variety  of  staff  positions  and  can  be  used  by  officers  at  home,  school,  and  armory. 
Currently  developed  and  distributed  modules  encompass  twelve  courses  for  battalion  staff 
members  (BN-BSTS),  and  twelve  courses  for  brigade  staff  members  (BDE-BSTS)  with  a 
separate  common  core  course  for  both  the  battalion  and  brigade  level.  Figure  1  presents  the 
course/lesson  flow  for  ITTBBST-BSTS  courses,  using  both  text  and  CBI  instruction.  Within 
BN/BDE-BSTS  when  instruction  included  complex  tasks  requiring  coordination,  synchronization, 
or  a  practical  application  of  the  skills  and  knowledge  being  presented,  CBI  was  selected  as  the 
presentation  methodology. 

The  design  of  the  course  and  lessons  allows  the  student  to  control  the  sequence  and  pace  of  the 
training.  An  introduction  is  provided  at  the  beginning  of  each  subject  and  the  student  can  select 
whether  or  not  he  or  she  wants  to  complete  the  course  for  credit.  If  “credit”  is  selected  the 
student  must  take  a  pretest  to  determine  starting  knowledge.  If  “credit”  is  not  selected,  a  student 
will  not  take  a  pretest,  and  will  not  be  allowed  to  take  the  lesson  exams.  The  student  is  then  able 
to  choose  the  desired  lesson.  Students  start  lessons  by  reviewing  the  introduction  contained  in 
CBI.  Where  appropriate,  the  student  is  referred  to  the  Student  Workbook  which  provides  the 
tasks,  conditions,  standards  and  performance  measures  for  the  lesson,  as  well  as  a  list  of  required 
reading  if  the  student  is  not  well  versed  in  doctrine  for  this  material.  Following  off-line  reading, 
which  includes  text-bped  lesson  material,  the  student  returns  to  CBI  to  complete  practical 
exercises,  study  tutorials,  and  complete  the  lesson  exam.  Cues  are  integrated  throughout  the 
instmctional  material  to  guide  the  student  between  text  and  CBI  material.  Various  navigation 
choices  and  options  are  available  to  the  student  in  the  CBI  material  that  allow  individuals  to  move 
as  quickly  as  desired  and  replay  information  as  needed.  If  a  student  passes  a  lesson  exam,  the 
information  is  recorded  and  the  student  selects  another  lesson.  In  the  event  of  a  failing  score,  the 
information  is  recorded  and  the  student  is  prompted  to  restudy  the  material.  For  selected  lessons, 
remediation  training  modules  are  included  in  CBI  and  would  be  available  to  the  student.  After 
further  study  the  student  should  retake  the  lesson  exam. 

ITTBBST-BSTS  courses  are  composed  of  subjects;  subjects  consist  of  groups  of  lessons, 
and  lessons  are  composed  of  topics  that  cover  a  major  learning  objective.  Lessons  normally  take 
one  to  two  hours  to  complete  for  an  officer  of  average  ability.  The  training  management  system 
developed  for  the  BSTS,  Environment  for  MultiMedia  interactive  instruction  ^MMii),  allows  the 
trainer  or  other  designated  individual  in  the  unit  or  school  to  monitor  the  progress  of  individual 
battle  staff  officers  as  they  proceed  through  the  courses  of  instruction  and  attain  mastery  of  their 
individual  skills  and  knowledge.  All  diagnostics,  performance  measurements,  and  feedback  are 
captured  in  the  EMt^i  data  base,  enabling  student  progress  and  status  reports  to  be  provided  to 
the  trainer.  Instmction  on  mastery  of  basic  skills  and  knowledge  is  followed  by  scenario-based 
application  of  critical  tasks  in  the  Comprehensive  Assessment  Component  (COMPS)  to  ensure  the 
student  can  apply  teaching  points  in  a  realistic  environment. 


3 


Figure  1.  Course/lesson  flow. 


Doctrinallv-Based  Performance  Taxonomy 

Given  the  task-driven  nature  of  Army  training,  tools  for  identifying,  structuring,  and 
organizing  tasks  critical  to  combat  effectiveness  are  essential.  In  a  major  ongoing  effort,  ARI  is 
analyzing  BFs  to  establish  a  common  basis  for  structuring  unit  training  programs  and  assessing 
unit  performance  (e.g.,  Harrison,  1995).  The  Army's  Battlefield  Operating  Systems  (BOS)  ^ 
established  in  the  Blueprint  of  the  Battlefield  (U.S.  Army  Training  and  Doctrine  Command  1991) 
provide  the  basic  structure  for  the  BFs. 

Analysis  has  resulted  in  a  detailed  inventory  of  the  components  necessary  to  guide 
planning  of  training  exercises  and  assess  unit  performance.  Extensive  task  analyses  of  BFs  have 
been  completed  at  the  battalion  task  force  (TF)  level  and  the  brigade  level.  In  addition,  task 


*  “Battlefield  Operating  System”  is  being  replaced  by  “the  Army  Unit  Training  List  (AUTL).” 
This  report  uses  BOS  to  more  clearly  relay  an  understood  intent  until  the  terminology  issue  is 
resolved. 
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analysis  of  fire  support  coordination,  integration  and  synchronization  at  levels  from  brigade 
through  corps  has  been  performed. 

The  BF  task  analyses  are  a  valuable  tool  for  ensuring  consistency  across  related  training 
development  efforts.  They  provide  common  structure  intended  to  facilitate  the  design  and 
implementation  of  TSPs  at  multiple  echelons.  The  methodology  supports  the  determination  of 
linkages  and  dependencies  across  functional  elements  of  a  brigade  or  battalion.  Results  help 
define  objective  measures  of  performance  based  on  observable  behavioral  outcomes  (Kemper, 
1995).  In  short,  BF  task  analyses  provide  a  basis  for  integrating  training  developments  addressed 
in  ITTBBST-BSTS  and  establishing  consistent  approaches  to  the  measurement  of  performance. 

Value  of  CBI 


In  his  article.  Advanced  Technologies  Applied  to  Training  Design:  What  Have  We 
Learned  About  Computer  Based  Instruction  in  Military  Training?,  Fletcher  presents  a  summary 
of  some  research  efforts.  “Dana  (1987)  reported  reductions  from  40  percent  to  10  percent  in 
washback  rates,  earlier  screening  of  student  suitability  for  training,  and  1-2  week  reductions  in 
traimng  time.  Yasutake  (1987)  reported  24  percent  to  35  percent  time  savings  for  four  courses, 
positive  (80-90% )  student  attitudes,  and  negative  instructor  attitudes  for  computer  managed 
instruction.  Noja  (1987)  reported  reductions  in  training  time  from  8  to  5  weeks,  equivalent 
student  achievement  for  electronic  theory  and  improved  student  achievement  for  electronic 
applications.  In  comparing  results  from  a  computerized,  hand-held  training  aid  with  text-based 
workbooks.  Wisher  (1987)  reported  more  course  completions  (91%  as  contrasted  with  58%)  and 
better  test  performance  by  a  ratio  of  2: 1  for  the  computerized  training  aid.  Noja  (1991)  reported 
30  percent  to  50  percent  reductions  in  training  time  and  per  student  per  year  savings  of  $5,500  for 
CBI.”  (Fletcher,  1995) 

Much  of  the  value  of  CBI  to  military  trainers  lies  in  the  areas  of  student  interest  and 
resource  management.  Student  interest  is  reflected  by  the  studies  referenced  above.  Resource 
management,  on  the  other  hand,  involves  two  areas  of  concern  to  a  military  trainer:  scheduling 
time  and  overhead  costs.  Because  CBI  is  essentially  a  self-paced  and  independent  study  vehicle,  it 
automatically  reduces  the  scheduling/conflict  resolution  requirements  inherent  in  classroom  and 
learning  center  training.  In  addition,  costs  and  time  lost  in  instructor  and  student  travel  are  either 
reduced  or  negated,  dependent  on  the  method  of  CBI  employed. 

Statement  of  the  Problem 

While  individual  staff  skills  are  taught  in  TRADOC  schools,  individual  staff  roles  and 
functions  in  battlefield  synchronization  is  not  emphasized.  The  problem  addressed  by  ITTBBST- 
BSTS  concerns  the  need  for  multimedia  TSPs  to  support  assessment  and  individual  training  of 
commanders  and  staffs  at  armored  and  mechanized  infantry  battalion  and  brigade  headquarters  to 
hone  individual  skills  prior  to  collective  training. 

Evaluation  of  training  reveals  that  most  command  post  exercises,  vehicles  for  staff 
individual  training  as  well  as  staff  group  training,  are  of  short  duration  and  not  driven  by  scenarios 
sufficient  to  prepare  brigade  and  battalion  staffs  or  commanders,  as  individuals,  for  operations  at 
the  NTC.  Although  Rosenberger  (1995)  and  others,  such  as  Battle  Command  Battle  Laboratory 
and  Center  for  Army  Lessons  Learned  publications,  criticize  brigade  and  battalion  commanders 
for  making  excuses  for  poor  performance  at  the  NTC,  many  hindrances  to  training  cited  by 
commanders  do  in  fact  exist,  and  do  detract  from  the  sort  of  training  that  would  develop  an 
effective  warfighting  team  from  a  brigade  or  battalion  staff 
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BSTS  Training  Program  Foundation  Documents 

The  Commander's  Battle  Staff  Handbook  Thompson,  &  Valentine,  1993)  was 

published  by  the  Infantry  Forces  Research  Unit,  ARI,  Fort  Benning,  Georgia.  Linking  individual 
tasks  to  collective  tasks,  it  also  serves  to  smooth  the  transition  between  individual  tasks.  It 
contains  current  information  on  the  missions,  functions,  and  techniques  used  by  the  commander 
and  staff  at  the  battalion  level. 

The  Brigade  Commander’s  Battle  Staff  Handbook  (Andre  &  Valentine,  1996)  was 
published  by  the  Infantry  Forces  Research  Unit,  ARI,  Fort  Benning,  Georgia.  This  book  serves 
the  same  function  as  the  Commander ’s  Battle  Staff  Handbook,  but  at  the  brigade  level.  It 
contains  current  information  on  the  missions,  functions,  and  techniques  used  by  the  brigade 
commander  and  staff 

The  recent  update  of  TRADOC  Regulation  350-70,  Training  Development  Management, 
Processes,  and  Products  (U.S.  Army  Trmning  and  Doctrine  Command,  1995),  was  used  as  the 
guide  to  SAT  in  the  analysis,  design,  development,  implementation  and  evaluation  of  the 
ITTBBST-BSTS  project.  It  includes  the  processes  and  products  involved  in,  and  resulting  from, 
the  front  end  analysis  (FEA)  as  well  as  all  five  phases  of  the  SAT  process. 

The  latest  draft  of  TRADOC  Pamphlet  350-10-2,  Multimedia  Courseware  Development 
Guide  (U.S.  Army  Training  and  Doctrine  Command,  1996),  provides  guidance  for  producing 
multimedia  courseware.  It  describes  how  to  plan,  design,  develop,  and  validate  multimedia 
courseware.  As  a  draft  document,  this  was  used  as  a  general  guide  throughout  this  project. 

Preparing  Instructional  Objectives  (Mager,  1975)  provided  the  framework  for  the 
construction  of  training  objectives.  The  structure  of  the  training  objective  also  pointed  directly  to, 
and  supported  the  construction  of,  criteria  test  items  by  which  successful  performance  could  be 
determined.  Mager  insisted  that  a  clear  understanding  of  where  the  student  is  going,  in  terms  of 
goals  and  performances,  was  essential  to  the  accomplishment  of  instructional  aims. 

Rosenberger  (1995)  highlighted  numerous  shortcomings  in  battalion  and  brigade 
command  and  staff  work  at  the  NTC.  He  identified  a  lack  of  synchronization  as  the  cause  of 
many  failed  attacks,  attributing  many  of  the  defeats  to  the  commander  and  staff  officers’  failure  to 
realize  what  was  happening  on  the  battlefield.  His  observations,  and  those  of  others,  contributed 
to  the  needs  assessment  which  was  a  vital  part  of  the  FEA. 

Key  to  the  development  of  the  required  TSPs  was  the  re\dew  and  application  of  current 
doctrinal  and  training  publications  in  the  form  of  field  manuals  (FMs).  The  complete  reference  list 
used  in  the  development  of  ITTBBST-BSTS  is  at  Appendix  A.  Also  see  Appendix  F,  Review  of 
Related  Research. 


Technical  Objective 

To  develop  and  formatively  evaluate  BSTS-based  brigade/battalion  commander  training 
courses,  to  include  COMPS  and  required  remediation;  enhance  existing  brigade/battalion  staff 
officer  trmning  courses  with  COMPS  and  necessary  remediation  components;  update  the  orders 
and  doctrine;  and  formatively  evaluate  the  updated  courses  of  instruction. 
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Scope  of  the  Project 

In  accordance  with  the  Army's  crawl-walk-run  model  of  training,  maneuver  unit  training 
begins  at  the  individual  level,  progresses  through  crew,  platoon,  and  company  levels,  then 
culminates  in  battalion  maneuver  exercises  (Department  of  the  Army,  1988,  1990).  As  the  level 
of  training  increases,  both  realism  and  complexity  increase,  leading  to  the  ultimate  training 
exercise  at  one  of  the  Army's  Combat  Training  Centers  (CTCs). 

Individual  training  of  commanders  and  staff  officers  follows  the  Army  model.  The  first 
step  (crawl)  is  to  teach  individual  staff  and  command  skills.  The  ITTBBST-  BSTS  project  was 
initiated  to  deliver  28  prototype  multimedia  individual-oriented  TSPs  for  training  the  brigade  and 
battalion  commanders  and  staffs  of  the  armored  force  (Table  1).  Supporting  the  TSPs  was  the 
delivery  of  the  student  guide,  trainer’s  guide  and  EMMii  to  create  a  complete  training  system  that 
can  be  implemented  by  the  Force  XXI  Training  Program  in  the  total  force. 


Table  1 

ITTBBST-BSTS  Training  Support  Packages 

Brigade 

Battalion 

1.  COMMANDER 

15. 

COMMANDER 

2.  EXECUTIVE  OFFICER  (XO) 

16. 

XO 

3.  S  1/CHAPLAIN 

17. 

SI 

4.  S2 

18. 

S2 

5.  S3 

19. 

S3 

6.  S4 

20. 

S4 

7.  S5 

21. 

S3  AIR 

8.  S3  AIR 

22. 

CHEMO 

9.  CHEMICAL  OFFICER  (CHEMO) 

23. 

SIGO 

10.  SIGNAL  OFFICER  (SIGO) 

24. 

FSO 

1 1 .  FIRE  SUPPORT  OFFICER  (FSO) 

25. 

ENGINEER  (ENG) 

12.  ASSISTANT  BRIGADE  ENGINEER 

26. 

CHAPLAIN 

13.  AIR  DEFENSE  OFFICER  (ADO) 

27. 

ADO 

14.  COMMON  CORE 

28. 

COMMON  CORE 

Organization  of  the  Report 

The  remainder  of  this  research  report  contains  the  following  sections: 

The  BSTS  Development  Methodology  -  A  detailed  description  of  the  development 
process  for  ITTBBST-BSTS,  including  an  overview,  a  discussion  of  the  FEA,  and  an  in-depth 
examination  of  the  development  of  new  courses. 

Formative  Evaluation  -  Both  updated  and  new  courses  went  through  a  formative 
evaluation  (FE)  process.  This  section  discusses  the  design  of  the  FE  process,  testing  procedures, 
and  revisions  of  the  TSPs  derived  fi’om  evaluations. 

Discussion  -  This  section  examines  the  uses  of  ITTBBST-BSTS  courses,  as  well  as 
implementation  considerations. 
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Lessons  Learned  -  The  lessons  learned  in  the  preparation  and  update  of  BSTS  courses  are 
examined  in  this  section,  including  program  design,  development  methods,  and  FE  procedures. 

Conclusions  and  Recommendations  -  Points  covered  in  this  section  concern  conclusions 
made  as  a  result  of  the  program  and  recommendations  for  future  research  and  development. 


BSTS  DEVELOPMENT  METHODOLOGY 
Project  Overview 

The  need  for  individual  training  for  armored  and  mechanized  infantry  battalion  and  brigade 
commanders  and  their  staffs  set  the  stage  for  this  project.  ITTBBST-BSTS  was  developed  by 
applying  the  SAT  process  (Figure  2)  to  the  design  and  development  of  multimedia  CBI  and  text 
lessons.  The  SAT  cycle  is  a  five  step  loop  of  analysis,  design,  development,  implementation,  and 
evaluation. 


Analyze 

The  initial  step  of  the  SAT  cycle  is  analysis.  TRADOC  Regulation  350-70  defines  analysis 
(the  FEA)  in  terms  of  the  job  analysis.  Figure  3  depicts  the  BSTS  analysis  and  design  model.  The 
FEA  done  for  ITTBBST-BSTS  is  consistent  with  this  definition,  and  each  job  was  analyzed  in 
light  of  the  tasks  required  of  the  incumbent  in  the  performance  of  their  warfighting  duties.  The 
jobs  analyzed,  as  directed  in  the  Statement  of  Work  (SOW),  were  the  key  battalion  and  brigade 
battle  staff  positions,  including  commanders.  To  these  26  positions  was  added  a  common  core  of 
critical  warfighting  skills  required  by  all  commanders  and  the  members  of  their  respective  staffs. 
Therefore,  the  analysis  resulted  in  the  design  and  development  of  28  prototype  courses  (Table  1) 
of  instruction.  The  intended  (target)  population  was  the  total  force.  Active  and  Reserve. 

The  analysis  included  a  review  of  the  existing  BN-  and  BDE-BSTS  Program 
Design/Critical  Tasks  (Andre  &  Salter,  1996  b  and  c  respectively),  BFs,  review  of  applicable 
doctrinal  publications,  review  of  the  Center  for  Army  Lessons  Learned  (CALL)  publications, 
interviews  with  TRADOC  SMEs,  interviews  with  and  reviews  of  the  output  by  current  and  past 
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Figure  3.  BSTS  analysis  and  design  model. 

observer/controllers  (0/Cs)  at  the  CTCs,  and  review  and  approval  by  the  contracting  officer’s 
representative  (COR).  The  BFs  served  as  a  litmus  test  for  individual  critical  staff  functions  at 
battalion  and  brigade  level;  each  function  was  analyzed  to  ensure  that  the  requisite  individual  staff 
skills  were  included  in  course  material.  Battalion  and  brigade  FEAs  used  the  output  of  the 
available  BF  task  analyses  to  ensure  that  critical  tasks  had  been  included  in  the  courses.  This 
served  to  help  unify  and  integrate  BF  task  analyses  with  BSTS.  The  resulting  job  analysis  led  to 
the  next  phase  of  SAT,  design. 


Design 

The  design  process  used  the  results  of  the  FEA  to  determine  the  tasks;  identify  which  ones 
were  critical;  produce  tasks,  conditions,  and  standards  (TCS);  develop  criteria  test  items;  prepare 
course  maps;  develop  course  strategies,  and  prepare  examinations.  The  critical  task  list  and  draft 
course  map  were  reviewed  by  the  Operations  Group  of  the  NTC,  TRADOC  SMEs,  and  (in  the 
case  of  the  two  commander’s  courses)  Battle  Command  Battle  Laboratory  (BCBL).  The  BSTS 
FEA  output  was  provided  to  the  SGT  project  team  for  similar  review.  The  process,  illustrated  in 
Figure  3,  appears  to  be  linear;  however,  like  SAT,  it  is  iterative.  Formative  evaluation  was 
initiated  early  in  the  design  phase,  and  continued  throughout  the  life  of  the  courses. 

Develop 

The  third  step  in  the  development  of  the  28  BSTS  TSPs  was  creation  of  course  material. 
This  process  (Figure  4)  was  based  on  the  analysis  and  design  completed  by  the  research  team. 
Input  to  the  development  of  TSPs  was  composed  of  the  FEA,  specific  training  objectives,  and 
course  map.  During  the  conduct  of  this  phase  of  the  development  methodology,  both  the  text- 
based  instruction  and  the  storyboards  (including  the  practical  exercises  (PEs)  and  exams  as  well  as 
the  audio  visual  (AV)  material)  that  lead  to  the  development  of  the  CBI  were  created  by 
Contractor  Military  Experts  (CMEs)  and  graphic  artists.  Following  the  development  of  text  and 
CBI,  a  quality  assurance  (QA)  review  was  completed  by  the  BSTS  team. 
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Commercial  off-the-shelf  (COTS)  tools  and  templates  compatible  with  both  the  BN-BSTS  and 
BDE-BSTS  courses  were  used  in  ITTBBST-BSTS  CBI  development.  Templates  helped  define 
learning  strategies  for  mastery  of  skills  and  knowledge,  techniques  for  question  and  answer 
sessions,  testing  criteria,  screen  design,  navigation  through  courses  of  instruction,  helps,  job  aids, 
and  screen  layout. 

CBI  authoring  was  accomplished  using  Icon  Author^^  2  software,  assuring  compatibility  with  all 
BSTS  program  courseware.  Icon  Author  is  COTS  software;  therefore,  the  source  code  of  BSTS 
courseware  is  available  to  anyone  to  whom  the  government  chooses  to  provide  it.  The  authoring 
process  for  the  project  included  the  general  application  of  templates  developed  for  previous  BSTS 
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Figure  4.  BSTS  development  model. 


programs  and  the  development  of  new  templates  that  included  the  requirements  for  COMPS, 
remediation,  and  security  of  COMPS. 

The  development  of  BSTS  courses  was  executed  in  three  parallel  tasks: 

1.  Development  of  the  battalion  and  brigade  commanders’  BSTS  courses,  with  COMPS 
and  remediation. 

2.  Development  of  the  COMPS  for  all  courses. 

3.  Enhancement  of  the  courses  through  the  application  of  updated  tactical  doctrine.  This 
was  supplemented  with  new  tactics,  techniques  and  procedures  (TTPs),  and  lessons  learned 
gleaned  from  recent  Army  publications  (e.g.  Center  for  Army  Lessons  Learned  documents)  as 


2  Icon  Author  Version  7.0,  November,  1995.  A  trademark  of  AimTech  Corporation,  20  Trafalgar 
Square,  Nashua,  NH  03063. 
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well  as  feedback  from  formative  evaluation  by  TRADOC  SMEs.  This  also  included  modification 
of  existing  practical  exercises  to  align  with  Simulation-Based  Mounted  Brigade  Training 
(SIMBART)  standard  operation  orders.  Development  of  remediation  for  revised  courses  was 
done  as  required. 


Implement 

Implementation  on  a  trial  basis  was  executed  at  the  BSTS  developmental  laboratory  at 
Skidgel  Hall,  Fort  Knox,  Kentucky.  Phase  II  Alpha  participants  (SMEs)  and  Phase  HI  Beta 
participants  (target  population  officers)  were  selected,  briefed  and  then  issued  prototype  versions 
of  the  material  a  normal  student  would  receive  when  taking  an  ITTBBST-BSTS  course.  These 
participants  provided  feedback  and  demographic  data  that  was  returned  to  course  developers  for 
evaluation. 


Evaluate 


Because  of  the  nature  of  the  ITTBBST-BSTS  project,  implementation  and  evaluation 
were  so  closely  interrelated  that  implementation  overlapped  with  evaluation.  Results  were 
evaluated  to  determine  the  efficiency  and  effectiveness  of  the  training  products  in  attaining  the 
goal  of  training  battalion  commanders,  brigade  commanders,  and  their  respective  battle  staffs  in 
individual  staff  and  command  skills.  The  FE  was  conducted  in  three  phases,  that  included: 

Phase  I  -  Alpha  testing  by  CMEs 

Phase  II  -  Alpha  testing  of  the  complete  (draft)  training  support  package  (TSP)  by  military 
SMEs  at  Fort  Knox.  (This  included  at  least  one  tester  for  each  course.  Testers  had  usually  served 
in  the  staff  position  for  the  course  they  were  evaluating,  had  participated  in  actual  combat  and/or 
CTC  rotations,  and  most  were  currently  assigned  as  instructors  at  TRADOC  schools.) 

Phase  in  -  Beta  testing  of  the  complete  (draft)  TSP  by  target  population  officers  at  Fort 
Knox.  (This  included  two  to  five  testers  for  each  course.  Most  testers  had  completed  an 
assignment  as  a  battalion  or  brigade  staff  officer;  were  captains,  majors,  or  lieutenant  colonels; 
and  were  branch  qualified  for  schooling.) 

Phase  II-III  FE  lesson/subject  examination  results  determined  the  necessity  for 
remediation  lessons. 

A  fourth  phase  was  added  to  the  FE  process,  review  of  the  TSP  by  a  COR  representative. 
Phase  IV  FE  was  added  at  the  request  of  the  COR  and  conducted  by  government  representatives 
at  Fort  Knox. 


FEA  for  ITTBBST-BSTS 

The  FEA  was  a  top-down  process  that  provided  the  foundation  for  the  design  of  training. 
The  FEA  yielded  the  answers  to  eight  important  questions.  These  questions  were:  Who  is  to  be 
trained?  What  are  they  to  be  trained  to  do?  When  are  they  to  be  trained?  Where  are  they  to  be 
trained?  How  are  they  to  be  trained?  Why  are  they  to  be  trained?  How  well  must  they  be 
trained?  By  whom  will  they  be  trained? 

During  FEA,  the  tasks  identified  for  inclusion  in  the  ITTBBST-BSTS  were  derived  from 
those  listed  in  the  program  design/critical  tasks  developed  for  the  BN-BSTS  (Andre  &  Salter, 
1995)  and  BDE-BSTS  (Andre  &  Salter,  1996).  A  terminal  learning  objective  was  prepared  for 
each  course,  then  individual  tasks  in  the  form  of  TCS  were  compiled  into  lessons.  Battalion  level 
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tasks  fell  out  of  the  brigade  tasks;  in  many  cases  they  were  identical  as  a  result  of  the  top-down 
process. 


Use  of  Battlefield  Functions  as  Underpinnings  of  FEA 

The  FEA  was  the  application  of  the  analysis  step  of  the  SAT  process  to  ITTBBST-BSTS. 
It  included  review  of  literature  applicable  to  the  development  of  BSTS,  external  input 
requirements  to  the  analysis,  training  requirements,  and  training  need. 

The  intention  of  the  FEA  was  twofold.  The  initial  analysis  was  a  job  analysis  (as  described 
in  TRADOC  Regulation  350-70)  to  determine  individual  tasks  in  which  to  train  staff  officers.  The 
FEA  then  processed  to  identification  of  the  interfaces  used  by  a  commander  or  staff  officer  to 
integrate  individual  tasks  and  BOS  responsibilities  with  other  staff  members,  producing  a  team. 
This  progression  from  individual  job  to  staff  integration  is  analogous  to  the  relationship  between 
BFs  and  BOSs,  wherein  the  39  BFs  are  combined  in  subgroups  of  tasks  required  to  proficiently 
utilize  the  7  BOSs. 

The  foundation  of  the  FEA  for  ITTBBST-BSTS  was  the  FEA  from  BN-BSTS  and  BDE- 
BSTS.  Current  Army  doctrine,  CTC  lessons  learned,  and  doctrine-related  publications  were 
utilized  as  primary  material  to  update  the  FEA.  Additional  input  came  from  analysis  of  BFs  18 
(Plan  for  Combat  Operations),  19  (Direct  and  Lead  Unit  During  Battle  Preparation),  and  20 
(Direct  and  Lead  Units  in  the  Execution  of  Battle).  These  BFs  were  incorporated  into  the 
foundation  of  the  commanders’  courses  and  used  in  the  update  of  existing  courses. 

The  BFs  were  cross-referenced  with  the  original  framework  of  BSTS.  Because  BFs  and 
BSTS  had  common  antecedents,  there  was  almost  total  agreement.  Terminal  learning  objectives 
and  TCS  were  refined  based  on  this  review  so  as  to  align  closely  with  BFs. 

Update  of  Existing  TCS/Courses 

In  each  course  the  student  is  taken  through  the  crawl-walk-run  model  of  training.  Students 
begin  with  specific  performance  measures,  proceed  through  the  terminal  learning  objective  of  each 
lesson,  and  emerge  with  an  understanding  of  the  BOSs  and  their  integration.  As  stated  earlier, 
terminal  learning  objectives  for  ITTBBST-BSTS  courses,  as  well  as  TCS  for  lessons,  can  be 
directly  linked  to  each  of  the  BFs.  This  means  that,  at  a  more  basic  level,  TCS  required  to  meet 
the  terminal  learning  objective  can  also  be  viewed  as  leading  towards  the  successful  performance 
of  the  BFs.  Since  TCS  identify  the  skills  the  individual  st^  member  must  master,  to  include  the 
synchronization  of  those  skills  with  the  unit  commander  and  the  rest  of  the  staff,  ITTBBST-BSTS 
is  a  viable  vehicle  to  address  the  battlefield  synchronization  problems  identified  by  Rosenberger 
(1995)  and  others. 

TCS  were  updated  through  use  of  the  new  generation  of  publications  emerging  since  the 
publication  of  FM  100-5  (Department  of  the  Army,  1993).  The  methodology  for  TCS  update 
was  specified  in  the  ITTBBST  Design  Document  (BDM  Federal,  Inc.,  1996).  On  review  of 
current  and  emer^ng  doctrine,  TCS  were  selected  for  update  and  other  tasks  were  identified  as 
required  to  be  trained.  Updates  were  prepared,  or  new  TCS  created.  Text  and  CBI  lessons  were 
then  modified  or  developed.  All  lessons  were  rewritten  to  incorporate  the  latest  doctrinal 
material. 


Creation  of  TCS  for  Battalion  and  Brigade  Commanders 

Professional  competence  of  the  commander  is  characterized  in  FM  100-5  as  a  significant 
part  of  any  unit’s  combat  power.  In  the  pamphlet  A  commander ’s  guide  for  the  coordination  and 
employment  of  battlefield  operating  systems,  the  BCBL  reported  that  battle  commanders  did  not 
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have  a  general  level  of  battle  command  competencies  (Battle  Command  Battle  Laboratory, 

1995).  As  a  direct  result  of  the  FEA  methodology  discussed  above,  ITTBBST-BSTS  courses 
were  identified  as  necessary  for  commanders  at  battalion  and  brigade  levels,  something  lacking 
from  previous  versions  ofBSTS.  Utilizing  the  steps  delineated  in  Table  5  of  the  ITTBBST 
Design  Document  (BDM  Federal,  Inc.,  1996),  both  courses  were  created  using  the  findings 
contained  in  the  BCBL  Commoner 's  Guide. 

The  TCS  were  designed  to  provide  basic  instruction  in  the  essential  elements  of  a 
commander’s  professional  knowledge.  They  identified  and  provided  a  framework  for  lessons  in 
the  decision  making  process,  the  components  of  battle  command  and  control  (C^),  the  functions 
of  each  of  the  conunander’s  primary  staff  members,  combat  support,  and  combat  service  support. 
In  essence,  the  commander’s  courses  were  designed  to  provide  basic  instruction  on  the  BOSs  and 
their  synchronization. 


Doctrinal  Updates  to  BN/BDE-BSTS 

FM  100-5,  Operations  (Department  of  the  Army,  1993),  was  written  to  reflect  Army 
thinking  in  an  era  where  the  strategic  equation  has  radically  changed  from  that  of  only  a  decade 
earlier.  Its  genesis  was  the  fact  that  the  Cold  War  was  over,  thus  the  nature  of  the  strategic  threat 
was  radically  different,  requiring  reexamination  of  doctrinal  solutions  to  that  threat.  The  new 
edition  of  FM  100-5  spawned  a  series  of  doctrinal  publications  to  develop  branch-specific  aspects 
of  Army  doctrine,  all  intended  as  capstone  manuals  implementing  FM  100-5.  In  addition,  BCBL 
publications  and  Student  Text  101-5  (U.  S.  Army  Command  and  General  Staff  College,  1995) 
identified  and  addressed  shortfalls  in  battle  command  competencies  and  the  decision  making 
process. 

The  new  publications  were  utilized  in  the  doctrinal  review  that  updated  BN/BDE  BSTS 
and  created  ITTBBST-BSTS.  The  BCBL  Commander’s  Guide,  FM  71-3,  and  FM  100-5  served 
as  the  focus  documents  for  course  development.  As  each  course  was  developed,  the  source 
material  utilized  reflected  the  branch-specific  adaptations  of  new  doctrine.  \^ere  lessons  were 
more  general,  FM  100-5  itself  served  as  the  basis.  Throughout,  the  BCBL  Commander’s  Guide 
was  utilized  as  a  framework  and  map  of  key  points  that  the  lessons  were  required  to  address. 

As  may  be  expected  during  a  time  of  quickly  evolving  doctrinal  change,  inconsistencies 
were  discovered  witlun  and  between  different  publications  addressing  the  same  area  from  different 
viewpoints.  In  many  cases,  this  was  simply  due  to  separate  manuals  with  overlapping  areas  being 
updated  over  extended  periods  by  several  agencies  with  interrelated  responsibilities.  When  these 
doctrinal  inconsistencies  were  uncovered,  most  were  resolved  through  review  of  the  most  recent 
publication  from  the  TRADOC  proponent.  In  other  cases,  conferring  with  the  responsible  agency 
and  discussing  the  inconsistency  led  to  a  satisfactory  conclusion. 
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The  Role  of  SIMBART  Operation  Orders 

Scenarios  were  previously  used  in  BSTS  courses  to  facilitate  PEs.  However,  these 
scenarios  were  not  the  ones  used  with  other  TSPs.  Therefore,  all  scenario  based  PEs  and 
instructional  references  were  updated  to  use  the  SIMBART  operation  orders.  By  design, 
SIMBART  TSPs  provided  scenarios  through  the  use  of  operation  orders  for  MTC,  DATK,  and 
DIS.  These  orders  were  utilized  to  create  scenarios  in  which  practical  exercises  pd  COMPS 
were  structured  so  as  to  give  the  student,  not  only  a  realistic  feel  for  the  BOS  being  taught,  but  an 
opportunity  to  integrate  and  synchronize  teaching  points  for  various  BOSs  examined  throughout 
the  course.  This  method  gave  the  student  instruction  in  both  areas  of  professional  competence 
and  systematic  management  of  the  information  learned. 

New  Courses  and  Course  Components 

Courses  for  the  armored  and  mechanized  infantry  brigade  and  battalion  commanders  were 
added  to  the  BSTS  library  based  on  the  identified  need  for  the  training  of  these  commanders  in 
their  individual  warfighting  tasks.  Some  officers  are  selected  for  command  that  have  had  a  recent 
assignment  in  a  non-troop  unit,  thus  lacking  in  current  experience.  This  issue,  coupled  with 
lessons  learned  from  unit  performance  at  the  CTCs,  formed  the  basis  for  the  need  for  the 
commander’s  courses.  Commander’s  courses  were  developed  to  be  used  prior  to  a  designated 
commander’s  attendance  at  the  Pre-Command  Course,  not  in  lieu  of  it.  A  COMPS  module  was 
added  to  each  course  to  assess  student  knowledge  at  the  end  of  the  course  and  their  ability  to 
synchronize  battlefield  operating  systems.  A  remediation  component  was  also  developed  for  each 
lesson  where  FE  criteria  showed  it  to  be  necessary. 

Commander’s  Courses 


Once  the  need  had  been  identified  for  new  courses  for  battalion  and  brigade  commanders, 
the  majority  of  the  development  process  was  basically  the  same  as  that  utilized  to  update  existing 
courses.  Added  to  this,  however,  were  requirements  for  new  areas  of  research  and  analysis,  not 
updating,  of  TCS  and  performance  measures.  The  following  paragraphs  outline  the  detailed 
process  used  to  develop  these  courses. 

Research/FEA  for  Commander’s  Courses 


Research  included  current  doctrinal  literature,  proponent  school  doctrinal  experts,  the 
BCBL  Commander ’s  Guide,  and  CALL  newsletters.  ^^Tien  using  doctrinal  materials,  approved 
or  final  doctrine  was  used,  as  this  should  be  available  to  the  training  population.  In  some  cases 
emerging  doctrine  or  draft  versions  were  used,  but  this  fact,  when  applicable,  was  noted  witWn 
the  body  of  the  training  material.  The  proponent  school  provided  the  current  status  of  doctrine 
development,  along  with  any  new  equipment  and  its  scheduled  fielding  plan.  Other  sources  of 
information  available  at  the  proponent  school  included  the  table  of  organization  and  equipment 
manager  and  the  training  development  section  or  directorate. 

The  components  of  battle  command  were  analyzed  to  determine  the  individual 
competencies  required  of  battle  commanders.  This  included  an  analysis  of  missions,  identification 
of  tasks  and  subtasks,  and  identification  of  conditions  under  which  they  are  performed.  Command 
and  control  competencies  were  addressed  on  two  levels;  the  global  level,  as  terminal  learning 
objectives,  and  at  the  individual  level,  as  TCS. 

Terminal  Learning  Objective  (TLO)  &  Task  Analysis 

The  TLO  was  developed  as  a  short  narrative  describing  the  purpose  of  a  particular  course 
of  instruction.  The  TLO  format  also  listed  TCS  for  each  lesson  within  the  course  of  instruction. 
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Performance  measures  were  published  in  the  instructional  material  to  provide  the  student  a  guide 
for  self-evaluation.  Task  analysis  was  directly  linked  to  construction  of  test  items. 


Course  Design/Outline 


The  SAT  process  (Figure  2)  ensured  the  systematic,  thorough,  and  orderly  construction  of 
the  courses  of  instruction,  and  the  continual  FE  of  each  course.  The  BSTS  development  model 
(Figure  4)  was  used  to  implement  the  SAT  process  in  course  development. 


Course  outlines  were  prepared  based  on  the  outcome  of  the  FEA  (Figure  5).  The  course 
outline  often  referred  to  as  the  course  map,  is  a  graphic  or  visual  portrayal  of  the  course  of 
instruction. 


BATTALION  COMMON  CORE 


Figure  5.  Sample/course  map 


The  courses  were  organized  by  subjects.  Within  each  subject,  lessons  to  be  taught  were 
identified.  Additionally,  each  lesson  was  prepared  to  correspond  to  a  task  or  tasks  previously 
identified  in  the  TLO.  The  lessons  were  fiirther  subdivided  into  topics  of  instruction  that  directly 
addressed  the  performance  measures. 

Text-Based  Lessons 

Using  the  established  performance  measures  for  the  lesson,  the  contractor  military  expert 
(CME)  prepared  a  draft  outline  presenting  the  instructional  material  in  a  logical  flow.  The  CME 
then  wrote  the  lesson  material,  referring  frequently  to  doctrinal  references.  Throughout  the 


wnting  process,  peer  review  and  assistance  was  a  common  practice,  especially  in  technical  areas 
where  another  CME  had  expertise.  (The  CME  team  members  totaled  more  than  170  years  of 
Army  service;  represented  combat  arms,  combat  support,  and  combat  service  support  branches; 
had  combat  as  well  as  CTC  experience  (observer/controllers,  OPFOR,  and  unit  rotations);  and 
completed  numerous  duty  assignments  as  battalion  and  brigade  staff  officers.) 

QA  review  was  constant  in  the  writing  process.  The  first  review  was  performed  by  the 
individual  writer,  checking  to  ensure  that  the  lesson  contained  instruction  for  each  of  the 
performance  measures,  presented  in  a  logical,  orderly  process.  Additional  checks  verified  use  of 
the  most  current  doctrinal  reference  material,  compliance  with  the  established  format,  and  that  the 
text  was  grammatically  correct. 

An  informal  review  was  conducted  by  peers  to  identify  any  doctrinal  inconsistencies  or 
ambi^ous  areas  in  the  text.  If  the  meaning  of  the  text  was  not  readily  understandable,  the 
material  was  rewritten  for  clarity.  The  course  development  manager  then  conducted  a  review 
focused  on  grammar,  adherence  to  established  format,  doctrinal  issues,  and  consistency  with  other 
courses. 

The  course  development  manager  and  instructional  systems  design  (ISD)  representative 
served  as  material  integrators.  The  ISD  representative  looked  to  see  if  the  text  met  identified 
performance  measures,  then  checked  the  work  for  information  flow  and  attention  to  detail.  After 
the  text  lesson  went  through  the  review  process,  appropriate  fixes  were  incorporated. 

Job  Aids 


In  addition  to  text-based  lessons,  training  support  materials  (job  aids)  were  developed. 
These  job  aids  were  designed  to  supplement  the  course  of  instruction,  and  provided  sample 
formats,  organizational  charts,  or  equipment  capabilities  data.  All  of  this  material  was  developed 
along  with  the  instructional  material  and  underwent  the  same  QA  process. 

Storyboards  (Computer-Based  Lessons! 

As  was  done  with  the  text  lessons,  the  computer-based  lessons  were  developed  (Figure  6) 
using  established  performance  measures.  In  some  cases  entire  lessons  were  selected  for  CBI  and 
in  other  instances  only  selected  topics  were  prepared.  Since  CBI  is  non-linear  in  progression,  the 
CME  prepared  a  draft  outline  that  included,  not  only  instructional  content,  but  also  a  proposed 
branching  structure  between  menus  and  instructional  modules. 

The  draft  outline  presented  the  instructional  material  in  a  logical  flow  and  served  as  the 
basis  for  a  production  meeting  between  the  course  development  and  CBI  teams.  This  meeting 
allowed  early  coordination  for  the  various  graphic  support  items  that  would  be  required  and 
suggestions  for  revising  the  course  structure.  The  CNffi  then  developed  instructional  material  by 
preparing  storyboards,  which  consisted  of  the  proposed  graphic  display  (slide,  photo,  video,  etc.) 
and  the  corresponding  narration.  Peer  review  and  assistance  during  development  was  used  in 
technical  areas  where  one  CME  had  specific  expertise. 

The  same  QA  review  as  outlined  earlier  for  text-based  lessons  was  used  for  computer- 
based  lessons.  Review  started  with  the  individual  Avriter,  progressed  through  peers,  the  course 
development  manager,  and  the  ISD  expert.  Once  programming  of  course  material  was  in 
progress,  there  was  continual  interface  between  the  CME  and  the  CBI  team. 
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Tutorials 


Based  on  task  analysis  and  existing  material  from  the  original  BSTS  courses,  some  lessons 
were  selected  to  be  100  percent  CBI.  These  lessons  presented  the  instruction  in  an  interactive 
lecture  type  mode,  where  the  student  progressed  through  the  instruction  using  the  computer  as  a 
trainer.  All  information  was  contained  in  screen  displays  and  narration.  There  were  numerous 
branching  options  and  navigation  possibilities  so  the  student  could  progress  at  a  personal  pace, 
skip  past  topics  that  had  already  been  mastered,  and  study  only  the  desired  sections.  While  the 
student  was  allowed  to  select  the  navigation  options  and  the  pace  at  which  the  material  was 
reviewed,  the  instruction  was  tutorial  in  nature. 


CBI  STORYBOARD  &  PROGRAM 
DEVELOPMENT 

PHASE  I:  INITIAL  SELECTION  &  DEVELOPMENT 


IDENTIFICATION 
OF  TASKS  TO  BE 
DEVELOPED  AS 
CBI 

CME  DEVELOPS 
INITIAL  STORY- 
BCARD,  SELECTS 
MEDIA  &  SCRIPT 

CME  &  ISD  REVIEW 
STORYBOARD  FOR 
EDUCATIONAL 
VALUE  &  MEDIA 

PHASE  11:  INITIAL  &  FOLLOWUP  PRODUCTION  MEETING 

INITIAL 

PRODUCTION 

MEETING 

CME,ISO.CBI 

CME  REVISES 
STORYBOARD 

1^ 

MIDTERM 

PRODUCTION 

MEETING 

CME, ISD,  CBI  TEAM 

CBI  TEAM 
REVISES  CBI 
MATERIALS 


PROGRAM 

AUTHORING 


PHASE  III:  INTERNAL  QUALITY  ASSURANCE 


ALPHA  TESTING 
BYCME&ISD 

CBI  TEAM 
REVISES 
PROGRAM 

CBI  TEAM 
PRODUCES 
CD-ROM  DISK 

PHASE  IV:  EXTERNAL  QUALITY 
ASSURANCE/FORMATIVE  EVALUATION 


BSTS  TEAM 

ALPHA  TESTING 
BYSME 

REVISES 

PROGRAM 

BETA  TESTING 
BYTARGET  POP 

BSTS  TEAM 
REVISES 
PROGRAM 


FINAL 

DELIVERY 


Figure  6.  CBI  development  process 


Practical  Exercises 


In  many  cases,  merely  reading  and  studying  a  specific  topic  did  not  necess^y  assist  the 
student  in  mastering  the  desired  skill.  To  help  the  student  learn,  practical  application  was  often 
appropriate.  All  practical  exercises  were  100  percent  CBI  and  required  extensive  interaction  from 
the  student.  In  most  cases,  the  exercises  were  driven  by  a  scenario  based  on  the  SIMBART 
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operation  orders.  The  exercises  placed  the  student  in  a  simulated  staff  officer’s  position.  The 
student  then  used  and  applied  the  skills  just  studied. 

Media  Selection 


As  mentioned  earlier,  part  of  task  analysis  involved  selecting  the  best  means  of  presenting 
the  course  material,  either  text  or  CBI.  For  text-based  lessons,  visual  aids  were  developed  and 
included  in  topics  where  the  material  could  be  better  understood  through  a  combination  of  visual 
and  text  information.  For  those  lessons  and  topics  that  were  to  be  computer-based,  a  more 
complex  media  selection  process  was  applied. 

Computer-based  lessons  and  practical  exercises  were  designed  to  assist  the  student  in 
learning  selected  topics,  generally  those  more  complex  or  technical  in  nature.  To  display  the 
information  in  a  simple  form,  various  visual  presentations  were  used.  For  less  complex  topics, 
word  charts  highlighting  key  ideas  were  the  visual  media.  Where  a  graphic  depiction  or  photo 
helped  simplify  a  point,  it  was  employed.  In  some  cases,  actual  video  footage  was  deemed 
appropriate  so  the  student  could  more  closely  relate  to  the  topic  being  presented.  The  intent  was 
to  display  the  presentation  media  that  best  assisted  the  student  in  comprehending  the  instructional 
material. 

Technical  specifications  for  the  student  computer  station  maintain  compatibility  across  all 
the  BSTS  programs  using  the  Military  Personal  Computer  standard  (MPC-2).  Bulletin  board  and 
“chat”  capabilities  are  available  to  BSTS  staff  officers  in  either  the  wide  area  network  (WAN) 
mode  or,  the  local  area  network  (LAN)  mode  installed  at  Fort  Knox.  Students  can  also  complete 
a  course  in  a  stand-alone  computer  mode  developed  under  ITTBBST-BSTS.  These 
communication  methods,  available  in  the  EMMii  software,  can  prepare  battle  staff  officers  for 
collective  tasks  that  are  at  the  heart  of  training  issues  uncovered  in  ARI  research. 

Lesson  and  Subject  Assessments 

The  established  BSTS  assessment  model  was  applied  throughout  the  two  commander’s 
courses.  At  the  beginning  of  every  subject,  the  student  was  given  the  option  of  taking  a  pre-test 
to  assist  in  identifying  areas  needing  the  most  training.  Before  starting  the  instructional  material, 
the  student  then  Imew  which  areas  of  performance  were  weak  so  as  to  be  able  to  review  the 
necessary  doctrinal  references.  The  pre-test  had  to  be  completed  in  order  to  access  the  lesson 
exams. 


After  completing  a  lesson,  the  student  was  directed  to  take  an  exam  that  addressed  the 
specific  course  material  just  studied.  The  questions  were  designed  to  evaluate  the  student’s 
comprehension  of  the  instructional  material  .  The  exams  were  constructed  so  as  to  be  knowledge 
based,  not  scenario  driven.  In  some  instances,  the  student  had  to  apply  learned  skills,  but  not  in  a 
simulated  tactical  situation.  Following  each  subject,  the  student  then  progressed  to  the  post-test. 
This  test  contained  the  same  questions  as  the  pre-test,  but  the  questions  were  presented  in  a 
random  sequence.  The  questions  addressed  performance  measures  from  all  lessons  within  the 
subject.  For  both  the  post-test  and  lesson  exams,  the  student  received  feedback  immediately  after 
answering  the  question.  The  feedback  told  the  student  whether  or  not  the  answer  was  correct, 
restated  the  correct  answer,  and  provided  additional  or  reinforcing  information  in  that  topic  area. 
This  technique  allowed  the  student  to  track  progress  throughout  the  test/exam,  identify  weak 
points,  then  concentrate  follow-up  study  in  the  needed  areas. 

Internal  review  of  all  tests  was  completed  just  as  on  other  instructional  material. 

Individual  CMEs  conducted  a  self-review,  followed  by  peers,  course  development  manager,  and 
ISD  edits.  Very  close  attention  was  given  to  ensure  that  all  performance  measures  were 
adequately  evaluated  on  the  appropriate  tests.  The  specific  wording  of  questions  was  carefully 
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examined  to  eliminate  possible  confusion  or  minor  inconsistencies  in  wording  between 
instructional  material  and  test  questions.  The  intent  of  the  exams  was  to  evaluate  and  reinforce 
learned  skills. 


COMPS 


The  COMPS  were  included  in  BSTS  to  provide  a  comprehensive  assessment  of  the 
student’s  ability  to  perform  critical  tasks,  acquired  in  the  related  course  of  instruction,  in  a  tactical 
scenario.  COMPS  were  based  on  critical  tasks  identified  in  the  FEA.  Unlike  the  knowledge- 
based  questions  used  in  the  pre-tests,  post-tests,  and  lesson  epminations,  COMPS  questions  were 
application-based.  Questions  were  based  on  the  NTC  scenarios  provided  in  the  Slh^ART 
operation  orders;  MTC,  DATK,  and  DIS.  All  questions  were  in  a  computer-based  instruction 
format,  standalone,  and  not  interlinked. 

The  COMPS  questions  were  designed  to  assess  all  performance  measures  for  the  related 
course.  COMPS  were  designed  and  developed  through  a  review  process  of  critical  tasks  and 
performance  measures  by  the  CME  and  BSTS  design  team.  All  questions  were  one  of  the 
following  three  forms: 

1.  True/False.  These  questions  were  constructed  so  as  to  have  the  correct  response,  noted 
in  parentheses  after  the  question,  and  include  feedback. 

2.  Multiple  Choice.  Multiple  choice  questions  had  the  distracters  listed  after  the  question. 
Each  response  was  designated  “Correct”  or  “Incorrect”  and  included  feedback.  Feedback  for 
incorrect  answers  included  stating  the  correct  answer.  Questions  were  designed  to  have  a 
maximum  of  five  possible  choices 

3.  Matching.  Matching  questions  had  a  maximum  of  7  responses  per  matching  group. 
Feedback  statements  were  provided  for  each  response. 

COMPS  were  developed  as  a  stand-alone  component  of  the  courseware.  The  design  of 
COMPS  provided  limited  security  through  the  use  of  hidden  files;  authoring  of  test  questions 
using  Icon  Author's  SmartObject  text;  random  ordering  of  questions;  and  the  random  ordering 
of  answers/distracters  within  multiple  choice  questions.  All  questions  were  permanently  marked 
by  a  unique,  randomly  generated  identification  number  consisting  of  seven  (7)  alpha-numeric 
characters  in  the  upper  comer  of  each  test  question  screen,  the  Question  Unique  Identification 
Number,  Test  (QUBTT).  The  QUINT  number  was  intended  to  permit  rapid  and  accurate 
identification  by  systems  management  personnel. 

COMPS  underwent  all  phases  of  FE  with  the  text  and  CBI  courseware.  QA  criteria  for 
COMPS  included:  spelling  and  grammar;  all  buttons  and  navigational  tools  working  properly; 
questions  properly  coded  for  the  correct  and  incorrect  responses;  results  properly  coded  to  link  to 
EMMii;  an  assigned  QUINT;  no  QUINT  duplication;  scores  properly  determined;  questions 
linked  to  the  portion  of  the  lessons  within  the  subject  and  course  from  which  they  were 
developed;  random  ordering  of  questions;  and  random  ordering  of  answers  and  distracters. 

Remediation  Component 

A  remediation  component  was  developed  for  each  BSTS  lesson  where  necessary.  The 
cutoff  score  set  for  lesson  examinations  was  80  percent,  10  percentage  points  higher  than  the 
current  TRADOC  standard  for  student  performance  in  a  proponent-offered  course  of  instruction. 
This  performance  level  was  judged  to  be  consistent  with  the  “job  entry  level”  for  each  position 
addressed  in  BSTS  courses.  When  a  Beta  test  participant  could  not  attain  an  80  percent  cutoff 
score  in  an  examination,  the  lesson  was  carefully  reviewed  to  judge  the  necessity  for  remediation. 
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First,  the  complexity  and  difficulty  of  the  lesson  material  was  examined  for  any  indication 
that  remedial  instruction  might  be  needed. 

Second,  the  original  BN-BSTS  and  BDE-BSTS  FE  data  were  analyzed  to  determine  those 
critical  tasks  that  target  population  officers  were  having  difficulty  mastering.  The  analysis 
considered  the  following  factors: 

1.  Adequacy  of  Beta  participant  qualification.  (Participants  were  occasionally  found  to  be 
unqualified  in  the  areas  they  were  evaluating.) 

2.  Time  spent  in  study  of  materials.  (In  a  number  of  instances,  participants  spent  either  a 
few  minutes  or  no  time  at  all  on  course  materials  prior  to  taking  the  lesson  exam.) 

3.  Percentage  of  improvement  from  pre-test  to  post  test.  (There  were  several  instances  of 
Beta  participants  scoring  lower  on  the  post  test  than  on  the  pre-test.) 

4.  Terminal  performance  as  measured  by  the  lesson  examination. 

Based  on  Beta  test  results  of  previously  developed  BN/BDE-BSTS  courses,  certain 
lessons  met  the  criteria  for  remediation,  but,  due  to  other  considerations  were  completely 
redesigned.  Beta  testing  of  the  revised  lessons  in  ITTBBST-BSTS  suggested  remediation  was 
not  necessary. 

Third,  as  the  courses  themselves  were  Phase  HI  Beta  tested,  feedback  from  students  and 
raw  scores  on  the  tests  pointed  to  areas  in  need  of  remediation. 

The  lessons  with  remediation  components,  based  on  ITTBBST-BSTS  and  previously 
conducted  BN/BDE-BSTS  test  results,  are  listed  in  Table  2. 


Table  2 

Courses  and  Component  Lessons  With  Remediation 

Course 

Lesson(s) 

Brigade  Air  Defense  Officer 

2A.  Integrated  Combat  Airspace 
Command  and  Control  (ICAC^) 

2B.  Airspace  Control  Measures 

Brigade  Chemical  Officer 

1  A.  Organization  and  Duties 

Battalion  Commander 

3G.  Aviation  Support 

4B.  Forward  Support  Battalion 

4C.  Task  Force  Support 

Battalion  S3  Air 

3A.  ICAC^ 

3B.  Airspace  Control  Measures 

Battalion  Chemical  Officer 

1  A.  Organization  and  Duties 

2A.  Avoidance  and  Protection 
Operations 
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Remediation  module  development  followed  the  same  procedures  as  those  previously 
outlined  for  course  development,  with  the  exception  that  they  were  developed  as  100  percent  CBI 
lessons.  Remediation  storyboard  development  followed  the  same  procedures  as  outlined  for  all 
courses. 

Following  QA  and  Alpha  testing  by  BSTS  personnel,  the  completed  remediation  module 
was  incorporated  into  EMMii  and  delivered  for  Phases  II-IV  of  FE. 

The  entry  method  into  remediation  is  shown  in  Figure  1  as  part  of  the  course/  lesson  flow. 
When  a  student  scores  less  than  80  percent  on  a  lesson  that  has  a  remediation  module,  the  lesson 
remediation  button  automatically  appears  on  the  CBI  lesson  menu.  This  affords  the  student  the 
opportunity  to  take  the  remediation  module.  When  the  student  achieves  an  acceptable  score, 
EMMii  is  updated  with  that  score. 


FORMATIVE  EVALUATION 
Design 

The  evaluation  method  chosen  for  ITTBBST-BSTS  purposes  was  FE.  As  outlined  in  the 
Development  Methodology  section,  a  progressive  three-phase  evaluation  model  was 
implemented:  (a)  internal  review  by  contract  team  members,  (b)  external  evaluation  by  military 
SMEs,  and  (c)  external  testing  by  target  population  representatives. 

Specifically,  FE  was  designed  to  determine  if: 

1 .  Training  can  be  implemented  as  designed. 

2.  Students  who  complete  a  course  can  be  assessed  in  a  comprehensive  manner. 

3.  Training  is  developed  at  the  correct  educational  level. 

4.  Instructional  materials  are  of  high  quality,  correctly  reflect  course  design  decisions, 
identify  training  objectives  and  performance  standards. 

« 

5.  Instruction  is  doctrinally  correct. 

6.  The  instruction  appropriately  illustrates  and  describes  the  material. 

7.  Students  performed  to  prescribed  standards  after  training,  and  improvement  in 
performance  can  be  measured. 

8.  The  student  critique  system  affords  the  student  the  opportunity  to  freely  make 
comments  about  instruction  and  administrative  support  matters. 

9.  The  instructional  environment  is  conducive  to  learning. 

10.  Student  learning  is  taking  place  through  opportunities  for  interaction,  student 
practice,  testing,  and  reviews. 

FE  was  also  used  to  determine  the  reliability  of  the  computer  hardware  and  software 
delivering  the  training,  as  well  as  its  ease  of  use  and  functionality. 
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Procedures 


The  FE  exercises  were  conducted  on  government  provided  hardware  compatible  with 
hardware  and  software  of  previous  BSTS  courseware.  For  external  FE,  the  courses  were 
integrated  into  the  training  management  system  and  placed  on  the  LAN  in  the  BSTS 
developmental  laboratory  in  Skidgel  Hall,  Fort  Knox.  Officer  participants  were  coordinated  by 
the  COR  and  selected  by  the  government.  Analysis  of  FE  results  served  as  the  basis  for  the 
revision  of  TSPs  to  ensure  accomplishment  of  course  terminal  learning  objectives. 

The  FE  of  the  28  BSTS  courses  included  evaluation  of  CBI,  text,  COMPS,  remediation, 
and  the  hardware  and  software  providing  the  operating  environment  for  the  computer-based 
courseware.  The  BSTS  team  executed  FE  in  three  phases: 

1 .  Phase  I  -  Alpha  testing  by  CMEs. 

2.  Phase  II  -  Alpha  testing  of  the  complete  (draft)  TSP  by  military  SMEs,  one  SME  per 

course. 

3.  Phase  III  -  BETA  testing  of  the  complete  (draft)  TSP  by  target  population  officers;  2-5 
participants  per  course. 

To  this  three-phase  plan  was  added  a  Phase  IV:  Review  of  the  final  TSP  under  the 
direction  of  the  COR  at  the  laboratory  in  Skidgel  Hall. 

The  28  TSPs  were  divided  into  seven  groups  of  courses  for  efficient  management  of  the 
process.  Prior  to  the  start  of  each  external  iteration,  participants  were  briefed  on  the  BSTS 
program  as  well  as  the  conduct  of  the  evaluation. 

During  the  conduct  of  external  FE,  BSTS  team  members  observed  the  evaluation  to 
record  observations,  and  were  available  to  assist  participants  with  hardware  and  software  issues. 
To  measure  the  effectiveness  of  courseware,  FE  data  collection  forms  were  prepared  by  the  BSTS 
team  and  approved  by  the  government  (Appendix  B). 

Troop  support  requirements  for  FE  activities  were  substantial.  In  excess  of  149 
participants  were  required  from  Fort  Knox  and  the  proponent  TRADOC  schools  for  Phases  II-IV. 
Constant  coordination  was  required  between  the  COR  and  BSTS  team  to  ensure  a  smooth 
execution  of  the  evaluations.  There  was  a  designated  FE  Coordinator  on  the  BSTS  team  to  plan 
and  coordinate  the  troop  support  requirements  with  the  COR  and  the  ARI  Armored  Forces 
Research  Unit  Research  and  Development  Coordinator.  These  personnel  integrated  BSTS 
requirements  with  the  requirements  for  the  other  ITTBBST  projects. 

Phase  I  -  Alpha  Testing  bv  CMEs 

The  BSTS  Team  conducted  internal  FE  as  delineated  in  the  training  development  models 
in  Figures  4  and  6.  At  each  of  the  analysis,  design  and  development  steps,  at  least  one  CME,  the 
team  manager/leader,  and  the  ISD  representative  conducted  Alpha  testing.  Phase  I  evaluation 
included  the  following: 

1.  Verification  that  the  comments  and  lessons  learned  provided  by  Armor  School  SMEs 
(from  previous  BSTS  efforts)  were  included  in  the  revised  TSP. 

2.  Adherence  to  the  FEA  in  the  training  of  critical  tasks. 

3.  Within  the  context  of  the  FEA,  confirmation  that  applicable  BFs  were  included. 
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4.  Confirmation  that  tests  (diagnostic  pre-tests,  lesson  exams  and  quizzes,  post  tests,  and 
the  COMPS)  accurately  measured  how  well  the  student  had  performed. 

5.  Adherence  to  standards  for  grammar,  punctuation,  and  format. 

6.  Correct  application  of  doctrine  to  include  the  most  current  doctrinal  reference;  tactics, 
techniques;  and  procedures  (TTP);  lessons  learned;  and  SME  input  from  the  CTCs. 

7.  Application  of  practical  exercises  for  critical  tasks  and  concepts  based  on  the 
SIMBART  operation  orders. 

Phase  II  -  Alpha  Testing  bv  Military  SMEs 

Revised  TSPs  were  presented  at  Fort  Knox  for  Phase  n  testing  by  the  government 
furnished  SMEs.  In  excess  of  30  SMEs  were  required  for  this  phase  of  evaluation.  This  phase 
sought  the  written  input  of  at  least  one  SME  for  each  course  of  instruction.  The  task  of  the 
SMEs  was  to  determine  the  doctrinal  accuracy  of  the  TSP,  whether  the  PEs  supported  the 
training  of  the  lesson  content,  and  if  the  tests  (pre-tests,  lesson  exams  and  quizzes,  post  tests  and 
COMPS)  reasonably  evaluated  the  subject  matter  that  was  trained.  Additionally,  SMEs  provided 
written  comments  on  the  computer  hardware,  software  and  the  training  management  system  under 
which  the  computer-based  instruction  operated.  Close  coordination  between  government  and 
contractor  personnel  ensured  timely  evaluation  and  delivery  of  TSPs. 

Phase  n  testing  results  were  collected  by  the  BSTS  team  members  on  site  at  Fort  Knox 
during  the  entire  13  month  testing  period,  then  returned  to  the  BSTS  development  team  for 
inclusion  into  the  revised  TSPs.  A  copy  of  the  Phase  11  test  results  with  recommended  actions  for 
enhancement  of  the  TSPs  was  provided  to  the  COR  for  review  and  comment. 

Phase  m  -  Beta  Testing  bv  Target  Population  Officers 

Third  generation  TSPs  were  delivered  to  Fort  Knox  for  Phase  III  testing  by  the 
government  furnished  target  population  officers.  In  excess  of  77  officers  from  the  target 
population  were  required  for  this  phase  of  evaluation.  Phase  III  testing  of  each  of  the  28  courses 
was  conducted  by  2-5  target  population  officers.  Target  population  personnel  were  selected  from 
the  following:  graduates  of  the  Officer  Basic  Course  (OBC),  OflScer  Advanced  Course  (OAC), 
battalion  and  brigade  command  designated  officers,  and  TRADOC  proponent  school  SMEs.  A 
description  of  the  target  population  for  each  of  the  BSTS  TSPs  is  provided  at  Appendix  C. 

Complete  TSPs  were  evaluated,  to  include:  the  text,  CBI,  examinations,  COMPS,  and  the 
corresponding  remediation  component.  Data  were  collected  to  ensure  that  participants  could 
meet  the  established  standards.  The  FE  procedures  included  direct  observation,  questionnaires, 
individual  interviews,  and  on-line  evaluation  and  feedback  from  the  participants. 

Phase  IV  -  Final  Review  bv  the  COR 

The  refined  TSPs  were  delivered  to  Fort  Knox  for  Phase  IV  review  by  the  COR  and 
government  furnished  personnel.  In  excess  of  28  personnel  were  required  for  this  phase  of 
evaluation.  Phase  IV  FE  was  conducted  to  ensure  that  all  planned  revisions  to  the  TSPs, 
contained  in  the  recommended  actions  section  of  the  Phase  III  report,  were  in  fact  included. 

Key  items  of  review  during  Phase  IV  included  completeness  of  revisions,  functionality  of 
all  navigational  buttons  in  CBI,  and  that  all  CBI  graphics  colors  were  correctly  displayed  on  the 
target  hardware. 
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Revision  of  Training  Products 

The  results  of  Phase  II,  HI,  and  IV  FE  testing  were  summarized  in  a  report  for  the  review 
pd  approval  of  the  COR.  An  example  of  a  Phase  III  (Beta)  report  is  at  Appendix  D.  Of  key 
importance  in  this  report  is  Enclosure  3,  Recommended  Actions.  Based  on  the  recommended 
actions  (included  in  all  Phase  II-IV  reports)  and  feedback  from  the  COR,  the  TSP  was  revised  and 
the  resulting  TSP  delivered  to  the  COR  for  subsequent  review,  culminating  in  the  final  COR 
review  (Phase  IV).  This  process  enabled  the  COR  to  view  the  raw  evaluation  data  from  the 
students  as  well  as  the  recommended  actions  to  be  taken. 

Results 

The  output  of  the  four-phase  FE  process  was  the  28  final  TSPs  specifically  developed  and 
evaluated  to  enhance  the  warfighting  skills  of  the  brigade  and  battalion  commanders  and  their 
respective  battle  staffs.  These  final  TSPs  consisted  of 294  hours  of  CBI  plus  341  hours  of  text, 
for  a  total  of  635  hours  of  instruction.  Text  was  presented  in  a  loose-leaf  notebook  and  the  CBI 
copied  onto  one  or  more  CD-ROMs. 

A  summary  of  selected  data  from  the  FE  process  is  contained 
in  Appendix  E. 


DISCUSSION 
Application  of  the  Products 

The  ITTBBST-BSTS  TSPs  were  developed  to  train  the  armored  force  brigade  and 
battalion  commanders  and  their  key  battle  staff  members  in  their  individual  warfi^ting  tasks.  The 
start  point  for  the  project  was  the  existing  26  TSPs  developed  under  the  BN-  and  BDE-  BSTS 
programs.  To  this  baseline  was  added  courses  for  the  brigade  and  battalion  commanders.  All  28 
TSPs  were  enhanced  \Ndth  newly  published  tactical  doctrine,  TTPs,  CALL  lessons  learned,  and 
input  from  CTC  0/Cs.  Additionally,  the  project  integrated  the  SIMBART  operation  orders  into 
PEs  and  the  end  of  course  COMPS.  This  enhancement  facilitates  the  use  of  the  TSPs  in  an 
integrated  training  strategy  from  individual  training  (BSTS),  through  staff  group  training  (SGT), 
to  collective  training  (SIMUTA,  SIMBART,  and  COBRAS  exercises). 

The  TSPs  are  designed  to  be  employed  in  a  TRADOC  school,  a  tactical  unit,  an 
installation  learning  center,  national  guard  armory,  reserve  training  center,  or  at  a  student  officer’s 
home.  In  addition  to  the  TSP,  a  486  processor-based  multimedia  computer  is  required  to  conduct 
the  training.  (A  listing  of  hardware  and  software  requirements  is  provided  at  Appendix  G  and  is 
included  on  each  of  the  CD  covers.)  All  required  training  materials  are  contained  in  the  course 
textbook  (student  guide)  and  the  accompanying  CD-ROMs  that  contain  all  required  courseware 
and  application  software,  with  one  major  exception.  Course  materials  contain  an  operation  order 
overlay,  but  not  a  map.  The  required  maps  may  not  be  readily  available  to  all  students  and  course 
proponents  would  do  well  to  consider  their  inclusion  in  the  TSP. 

The  target  population  officer  (Appendix  C)  is,  depending  on  assigned  staff  position,  a 
branch  qualified  graduate  of  the  respective  branch  OBC,  OAC,  Combined  Arms  and  Service  Staff 
School  (CAS^),  or  Command  and  General  St^  College  (CGSC)  that  has  been  assigned,  or  is 
pending  assignment,  to  an  armored  force  battalion  or  brigade.  An  officer  should  take  the 
appropriate  course  as  early  as  possible  on  notification  of  assignment.  In  addition,  the  courses  are 
available  as  needed  for  refresher  or  cross  training,  as  desired.  Courses  are  designed  so  that  a 
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student’s  first  use  of  ITTBBST-BSTS  should  include  the  Common  Core  course.  After  that,  the 
student  is  prepared  to  take  any  specialty  course  desired. 

Review  of  FE  feedback  has  led  to  several  considerations.  Students  must  approach  the 
course  material  with  a  desire  to  learn  and  improve  their  skills.  Trainers  and  commanders  should 
not  use  exam  scores  to  “evaluate/rate”  their  staff  personnel.  BSTS  needs  to  be  used  as  a  teaching 
and  training  vehicle,  not  a  means  to  score  performance,  which  follows  the  same  philosophy  as 
rotation  results  fi'om  any  actual  CTC. 

Implementation  Considerations 

The  TSPs  have  been  designed  for  the  total  force,  active  and  reserve,  assuming  the 
organization  has  a  multimedia  computer  with  the  specifications  provided  at  Appendix  G. 
Implementing  FM  25-100  (pepartment  of  the  Army,  1988)  as  well  as  FM  25-101  (Department  of 
the  Army,  1990),  the  TSPs  are  meant  to  be  executed  under  the  tutelage  of  a  trainer,  using  the 
Trainer’s  Guide  to  assist  in  implementation.  However,  students  with  access  to  the  proper 
computer  equipment  may  take  the  course  on  their  own. 

The  TSPs  can  be  implemented  in  a  LAN,  WAN  or  stand-alone  computer  configuration 
with  a  CD-ROM  capability.  Figure  7  shows  a  model  of  the  network  implementation.  The  data 
generated  by  a  student  completing  a  TSP  are  recorded  in  the  EMMii  data  base  in  the  LAN  mode 
and  can  be  uploaded  to  the  data  base  from  WAN  and  stand-alone  mode. 


25 


Within  theUnitnKAOOC  SchooliLeaming  Center 


WAN 


STAND  ALONE 


Figure  7.  Network  implementation  model. 


A  training  management  system  (TMS)  administrator  has  access,  through  the  Training 
Information  System  (TIS)  database,  to  student  records  that  include  not  only  the  student’s 
performance  on  the  TSP  but  data  collected  to  assist  training  developers  in  maintaining  doctrinal 
accuracy.  Detailed  information  is  maintained  on  student  comments,  performance  on  examinations 
(including  statistical  analysis  of  each  question  in  each  examination),  and  the  courseware. 

Test  Participant  Reactions 

The  FE  results  indicate  that  staff  officers  are  receptive  to  new  training  techniques.  Based 
on  participant  ratings,  CBI  and  text  lessons  were  much  preferred  over  reading  standard  Army 
doctrinal  references.  There  are  some  fundamental  issues  that  should  be  considered,  however,  as 
the  Army  transitions  to  CBI  or  various  forms  of  distance  learning.  Provided  below  are  various 
observations  gleaned  from  the  FE  during  this  project. 
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As  is  true  with  any  self-paced  instruction,  the  student  must  be  motivated  to  learn  and 
proper  supervision  must  be  applied.  Analysis  of  Beta  test  data  indicated  exam  scores  below 
expected  standards  of  performance,  but  demographic  analysis  of  participants  provided  possible 
reasons.  Only  a  minority  of  Beta  participants  were  actually  officers  that  fit  the  profile  of  those  for 
whom  the  course  was  intended.  In  a  random  sample  of  30  participants,  for  example,  only  8  were 
in  the  target  population.  There  is  a  reasonable  expectation  that  a  target  population  officer  actually 
being  transferred  into  a  brigade  or  battalion  staff  would  have  a  greater  motivation  to  use  the 
course  in  the  manner  originally  intended.  Once  BSTS  is  implemented,  an  officer  motivated  to 
obtain  the  courseware  should  dedicate  the  time  required  to  study  the  material.  Officers  directed 
into  the  course  by  superiors  or  school  instructors  must  be  provided  the  needed  time  and 
supervisory  assistance. 

Based  on  accumulated  survey  data,  participants  spent  an  average  of  only  2  hours  reading 
doctrinal  references  for  each  of  the  battalion  courses  and  an  average  of  4  hours  for  each  of  the 
brigade  courses.  This  is  significantly  below  the  estimated  average  of  65  hours  needed  to  review 
all  doctrinal  references  for  a  course.  In  addition,  the  average  participant  spent  about  8  hours  less 
per  course  reading  the  text-based  lessons  than  was  projected.  This  could  support  the  h)q)othesis 
that  participants  read  through  the  text  lessons  quicWy  for  short-term  retention  (indicated  by  lesson 
exam  scores)  and  scored  lower  on  the  COMPS  which  required  longer  term  retention  of 
information.  Trainers  using  BSTS  courses  should  ensure  sufficient  time  is  allocated  so  officers 
can  properly  review  the  material  required. 

COMPS  questions  that  required  a  student  to  do  other  than  repeat  data  (for  example, 
analyze  orders,  evaluate  scheme  of  maneuver,  compare  with  commander’s  intent)  were  frequently 
assessed  as  judgmental  in  participant  feedback.  While  practically  all  scenario-based  questions  are 
situation  dependent,  the  fundamental  information  provided  in  the  scenario  is  sufficient  to  lead  a 
student  to  a  logical  conclusion/answer.  Based  on  assessment  of  student  comments,  students 
seemed  to  expect  perfect  information  and  clear-cut  situations  that  did  not  require  analysis, 
interpretation,  and  assessment.  In  real  situations,  information  is  frequently  incomplete  and  often 
confusing.  The  intention  of  the  COMPS  questions  was  to  require  the  participant  to  conduct 
analysis  in  order  to  arrive  at  a  solution. 

Participants  rated  the  CBI  and  tailored  text  nearly  two  to  one  over  doctrinal  references  for 
their  educational/training  value.  Participants  also  generally  preferred  scenario-based  and  practical 
application  exam  questions.  This  indicates  that  interactive,  scenario  driven  CBI  may  be  a  valuable 
method  for  maintaining  an  active  interest  in  the  material  by  students. 

LESSONS  LEARNED 

During  TSP  preparation,  a  number  of  issues  were  identified  which  represented  a 
significant  impact  (cost,  schedule,  or  both)  on  the  program  plan.  Lessons  learned  from  the  design, 
development,  and  evaluation  of  ITTBBBST-BSTS  courses  are  provided  in  this  section. 

Program  Design 

Throughout  the  development  of  these  courses,  numerous  doctrinal  manuals  were  revised 
and  published,  mandating  modification  of  course  content.  There  should  be  a  provision  in  the 
program  initiating  document  that  allows  each  item  of  new  material  to  be  expeditiously  assessed 
jointly  by  the  government  and  the  contractor.  Cut-off  dates  for  reference  material  should  be 
established  and  enforced.  There  is  a  point  at  which  the  expense  and  effort  of  including  newly 
developed  data  in  a  course  approach  diminishing  returns,  and  that  point  should  be  set  early  in  the 
process. 


27 


It  is  essential  that  the  government  be  an  active  participant  and  reviewer  of  the  FEA.  The 
FEA  will  drive  the  entire  course  design,  development,  and  content.  The  government  can  use 
contractor  expertise  to  develop  the  draft  FEA  and  course  outline,  but  government  review  and 
expeditious  approval  is  imperative.  Modifications  of  course  design  and  content  after  the  course  is 
under  development  can  lead  to  duplication  of  effort  and  wasted  time. 

The  government  should  provide  the  most  current  doctrinal  and  reference  material  for 
course  development.  This  would  ensure  that  the  government  has  control  over  the  material  being 
used  as  the  foundation  for  the  course,  and  could  make  assessments  of  what  material  to  include 
once  the  project  has  started.  It  also  simplifies  the  process  of  obtaining  doctrinal  references. 

Development  Methodology 

As  key  sections  of  lessons  were  complete,  and  in  some  cases  as  they  were  being  written, 
course  developers  would  consult  with  other  developers  for  their  review  and  comment.  This 
facilitated  consistency  of  doctrinal  information  between  courses  and  improved  course  structure. 
Concurrent  peer  review  by  other  CMEs  slowed  the  initial  development  process  to  some  degree, 
but  the  advantages  outweighed  the  time  spent.  Conducting  peer  review  throughout  the 
development  phase  and  omitting  it  at  the  end  of  lesson  development  still  allowed  lesson  material 
to  be  produced  on  time,  but  with  an  increased  information  interchange  and  improved  quality.  A 
final  overall  review  was  conducted  by  the  course  development  manager. 

Individual  CMEs  were  assigned  as  the  resident  experts  in  particular  subject  areas.  They 
studied  new  material,  provided  a  summary  to  other  CMEs,  and  became  the  focal  point  for  issues 
in  that  arena.  This  saved  some  time  and  precluded  the  requirement  for  eveiy  CME  to  review  all 
new  material,  while  ensuring  a  degree  of  consistency  between  courses.  This  technique  facilitated 
cross-leveling  of  information  in  a  fast-paced  development  process. 

If  it  became  available  in  the  course  development  process,  photo  or  video  media  were 
incorporated  in  courses  still  being  developed.  In  addition,  this  material  was  retrofitted  into 
previously  developed  courses  wherever  possible  and  appropriate.  While  this  effort  required 
additional  contractor  resources,  it  helped  ensure  the  best  available  material  was  included  in  the 
courses.  Copyrighted  music  segments  are  included  in  some  of  the  subject  introductions.  Written 
approval  fi’om  the  company  holding  the  rights  to  this  music  is  pending. 

Developing  prototype  trmning  materials  using  innovative  techniques  and  technology  is  a 
complicated,  time  consuming  process,  exacerbated  by  evolving  doctrine  and  resultant 
identification  of  new  requirements.  Contractor  staffing  and  scheduling  need  to  support  thorough 
internal  operational  testing  before  delivering  TSPs  for  external  testing,  with  the  flexibility  to 
accommodate  changes. 

The  equipment,  software,  and  authoring  tools  used  for  this  project  were  adequate.  As 
more  video  and  graphic  material  were  incorporated  into  the  CBI,  maximum  capacity  of  the 
computers  was  reached  on  many  occasions.  To  make  maximum  use  of  available  time  and  ensure 
the  highest  quality  multimedia  material,  the  equipment  used  by  CMEs  and  programming  authors 
should  be  upgraded  for  future  efforts. 

In  a  heavy  usage  situation.  Army  customers  (units  or  schools)  should  be  prepared  to 
replace  CD-ROM  drives  on  a  regular  basis.  Those  drives  that  do  not  use  a  caddie  are  generally 
more  reliable  than  those  that  do. 
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Formative  Evaluation 


During  FE,  statistical  data  were  compiled  on  course  material  in  order  to  make  effective 
judgments  on  content  and  presentation.  In  addition,  demographic  data  were  gathered  on  Alpha 
and  Beta  participants  in  order  to  assess  the  validity  of  their  replies  and  the  effectiveness  of  the  FE 
process  itself  Both  types  of  information  were  essential  in  order  to  properly  assess  comments  and 
determine  what  modifications  to  instructional  material  were  needed. 

Several  Beta  testers,  were  not  in  the  target  population  for  the  course  they  evaluated. 

Beta  testing  should  be  accomplished  by  actual  target  population  personnel.  Testing  by  other  than 
target  population  officers  leads  to  inconclusive,  and  occasionally  inaccurate  results  concerning 
whether  or  not  the  material  accomplishes  its  intent.  In  addition,  non-target  population  personnel 
completing  the  course  were  less  likely  to  possess  the  requisite  experience  and  skills.  TWs  could 
lead  to  an  inaccurate  conclusion  that  remediation  training  was  necessary,  simply  because  a  non¬ 
branch  qualified  or  inexperienced  officer  didn’t  have  the  requisite  background. 

Review  of  participant  feedback  during  FEs  was  not  performed  adequately  at  the  testing 
site.  Participant  comments  were  often  illegible  and  difficult  to  read,  and  frequently  very  unclear  in 
intent.  Some  forms  were  not  properly  completed,  which  meant  required  data  had  to  be 
extrapolated  from  available  information.  Tliis  required  the  CMEs  to  “interpret”  comments  and 
intent  in  order  to  make  suggested  modifications  to  course  material.  Both  the  government  agency 
supervising  the  test  participants  and  the  contractor  must  ensure  that  all  participants  are  fully 
briefed  on  their  responsibilities  and  debriefed  at  the  conclusion  of  their  participation.  Throughout 
testing,  periodic  checks  must  be  conducted  on  all  participants  to  ensure  compliance.  On-site  team 
members  experimented  with  entering  conunents  into  a  computer  database,  and  this  might  be 
pursued  in  future  efforts.  In  addition,  providing  a  course  developer  at  the  testing  site  could 
resolve  some  participant  issues  during  testing. 


CONCLUSIONS  AND  RECOMMENDATIONS 
Conclusions 

For  each  subject  in  every  course,  the  participant  was  required  to  take  a  pretest  before 
beginning  the  instructional  material  and  a  post  test  at  the  conclusion.  The  post  test  contained  the 
same  questions  as  the  pretest,  but  they  were  presented  in  a  scrambled  sequence.  Based  on  the 
scores  for  all  courses  across  all  Beta  participants,  the  average  improvement  in  test  scores,  from 
pretest  to  post  test,  was  more  than  20  percent. 

Each  individual  lesson  was  evaluated  independently  to  ascertain  that  student  average  exam 
scores  met  or  exceeded  the  established  80  percent  standard.  For  lessons  where  the  standard  was 
not  achieved,  remediation  modules  were  developed.  The  overall  average  lesson  exam  score  for  all 
battalion  level  courses,  based  on  more  than  250  scores,  was  87  percent.  The  overall  average 
lesson  exam  score  for  all  brigade  level  courses,  based  on  more  than  260  scores,  was  89  percent. 

Some  lessons  have  a  remediation  component  for  students  that  have  difficulty  learning  the 
material  from  the  basic  lesson.  Participant  scores  increased  about  15%  from  the  original  talang  of 
the  lesson  exam  to  the  exam  score  following  completion  of  the  remediation  component.  This 
apparent  positive  effect  of  remediation  may  be  skewed  due  to  the  fact  that  less  than  a  third  of 
participants  were  from  the  target  population,  and  those  participants  from  the  target  population 
showed  no  need  for  remediation. 
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The  COMPS  were  the  only  course  component  that  forced  participants  to  fully  use 
synchronization  skills,  one  of  the  basic  goals  of  the  ITTBBST-BSTS  program.  This  aspect  made 
them  extremely  valuable  from  an  instructional  standpoint. 

Continuous  FE  is  essential  throughout  the  development  program.  Active  participation  by 
users,  as  well  as  the  CME  peer  review  process,  markedly  improved  the  product. 

Recommendations 

The  development  schedule  needs  to  provide  sufficient  time  to  incorporate  revisions  to 
course  material  following  Alpha  testing.  When  developing  new  courses,  more  alterations  should 
be  anticipated,  and  hence  more  time  to  incorporate  changes  after  Alpha  testing. 

Feedback  fi’om  participants  and  test  results  indicate  that  BSTS  courses  are  valuable  TSPs. 
Course  materials  should  be  reproduced  and  distributed  throughout  the  Army,  for  use  in  schools 
and  units. 

To  increase  the  probability  of  student  learning  and  decrease  the  possibility  of  test  question 
compromise,  pretests,  post  tests,  and  lesson  exams  could  be  rewritten.  True/false  questions  could 
be  replaced  with  multiple  choice  questions.  Additional  questions  could  be  developed  so  pretests 
and  post  tests  contain  different  questions.  Each  exam  could  consist  of  a  pool  of  questions  so  each 
time  the  exam  is  taken,  it  would  present  a  random  selection  of  questions  fi’om  those  available. 

Course  developer  representatives  should  be  present  on  site  during  Alpha/Beta  FE  to 
capture  data  through  interviews  in  addition  to  participant-prepared  surveys.  This  shift  in  data 
collection  methods,  while  more  time-consuming,  would  produce  more  uniform  quality  of 
feedback,  a  better  understanding  of  participant  concerns  about  the  courses,  and  reduced  time 
requirements  for  developers  to  analyze  participant  responses  and  prepare  course  fixes. 

While  the  latest  draft  of  TRADOC  Pamphlet  350-70-2,  Multi-media  Courseware 
Development  Guide  (U.S.  Army  Training  and  Doctrine  Command,  1996),  provides  some  detailed 
guidance  for  producing  multimedia  courseware,  it  should  be  reviewed  and  updated  by 
organizations  that  have  current  CBI  development  experience. 

All  training  materials  must  be  evaluated  periodically  to  ensure  they  satisfy  training 
requirements  and  contain  current,  accurate  information  based  on  the  latest  doctrine  and  available 
equipment.  BSTS  courses  must  be  subjected  to  evaluation  and  updated  as  necessary.  The 
government  must  establish  the  procedures  and  capability  to  maintain  this  courseware,  either  with 
internal  assets  or  through  external  contract. 

Because  of  the  rapidly  growing  array  of  multimedia  capabilities,  programs  should  be 
developed  for  MPC3  compliant  computers  or  better.  MPC2  is  not  adequate  to  make  fiill  use  of 
the  BSTS  software.  Courses  for  other  staff  officers  in  other  types  of  units  and  organizational 
staff  levels  should  be  developed.  Based  on  lessons  learned  and  student  suggestions,  there  are 
various  innovations  that  could  enhance  the  courses.  Further  research  into  emerging  state-of-the- 
art  technologies  and  techniques  is  in  order. 
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ACRONYMS  AND  ABBREVIATIONS 


ADO 

ARI 

ASAT 

ATSC 

AUTL 

BBS 

BCBL 

BDE 

BDE-BSTS 

BF 

BN 

BN-BSTS 

BOS 

BSHB 

BSTS 

CAS^ 

CALL 

CBI 

CCF 

CD-ROM 

CGSC 

CHEMO 

CME 

COBRAS 

COMPS 

COR 

COTS 

CSS-BSTS 

C/ST 

CTC 

DATK 

DIS 

DO 

EMMii 

ENG 

FE 

FEA 

FM 

FSO 

FXXITP 

HumRRO 

ISD 

ITTBBST 

LAN 

MOP 

MPC 

MTC 

NTC 

OAC 

OBC 


Air  Defense  Officer 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

Automated  Systems  Approach  to  Training 

Army  Training  Support  Center 

Army  Unit  Training  List 

Brigade/Battalion  Battle  Simulation 

Battle  Command  Battle  Lab 

Brigade 

Brigade  Battle  Staff  Training  System 

Battlefield  Function 

Battalion 

Battalion/Task  Force  Battle  Staff  Training  System 

Battlefield  Operating  System 

Battle  Staff  Handbook 

Battle  Staff  Training  System 

Combined  Arms  and  Services  Staff  School 

Command  and  Control 

Center  for  Army  Lessons  Learned 

Computer-Based  Instruction 

Critical  Combat  Function 

Compact  Disc-Read  Only  Memory 

Command  and  General  Staff  College 

Chemical  Officer 

Contractor  Military  Expert 

Combined- Arms  Operations  at  Brigade  Level,  Realistically  Achieved  through 
Simulation 

Comprehensive  Assessment  Component 
Contracting  Officer's  Representative 
Commercial  Off-the- Shelf 

Combat  Service  Support/Battle  Staff  Training  System 

Commander/Staff  T  rainer 

Combat  Training  Center 

Deliberate  Attack 

Defend  in  Sector 

Delivery  Order 

Environment  for  Multi-Media  Interactive  Instruction 
Engineer 

Formative  Evaluation 

Front-End  Analysis 

Field  Manual 

Fire  Support  Officer 

Force  XXI  Training  Program 

Human  Resources  Research  Organization 

Instructional  Systems  Design 

Innovative  Tools  and  Techniques  for  Brigade  and  Below  Staff  Training 

Local  Area  Network 

Measure  of  Performance 

Military  Personal  Computer 

Movement  to  Contact 

National  Training  Center 

Officer  Advanced  Course 

Officer  Basic  Course 
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0/C 

OPORD 

PE 

QA 

QUINT 

51 

52 

53 

54 

55 
SAT 
SGT 
SIGO 
SIMBART 
SIMITAR 
SIMNET 
SIMUTA 
SME 
SOW 
SGT 
TADSS 
TCS 

TD 

TF 

TIS 

TLO 

TMS 

TRADOC 

TSP 

TTP 

WAN 

xo 


Observer/Controller 
Operation  Order 
Practical  Exercise 
Quality  Assurance 

Question  Unique  Identification  Number,  Test 
Administrative  Officer  (battalion  or  brigade  staff) 
Intelligence  Officer  (battalion  or  brigade  staff) 

Operations  and  Training  Officer  (battalion  or  brigade  staff) 
Logistics  Officer  (battalion  or  brigade  staff) 

Civil  Affairs  Officer  (battalion  or  brigade  staff) 

Systems  Approach  to  Training 
Staff  Group  Trainer 
Signal  Officer 

Simulation-Based  Mounted  Brigade  Trjuning 
Simulation  In  Training  for  Advanced  Readiness 
Simulation  Networking 

Simulation-Based  Multiechelon  Training  for  Armor  Units 
Subject  Matter  Expert 
Statement  of  Work 
Staff  Group  Trainer 

Training  Aids,  Devices,  Simulators  and  Simulations 
Task,  Condition,  and  Standards 
Training  Development 
Task  Force 

Training  Information  System 

Terminal  Learning  Objective 

Training  Management  System 

U.S.  Army  Training  and  Doctrine  Command 

Training  Support  Package 

Tactics,  Techniques,  and  Procedures 

Wide  Area  Network 

Executive  Officer 
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APPENDIX  A 


BSTS  Doctrinal  Sources  List 
Field  Manuals 


FM  1-100  Doctrinal  Principles  for  Army  Aviation  in  Combat  Operations 
FM  1-111  Aviation  Brigades 

FM  1-112  Tactics,  Techniques,  and  Procedures  for  the  Attack  Helicopter  Battalion 

FM  3-3  Chemical  and  Biological  Contamination  Avoidance 

FM  3-3-1  Nuclear  Contamination  Avoidance 

FM  3-4  NBC  Protection 

FM  3-5  NBC  Decontamination 

TM3-7  NBC  Handbook 

FM  3-19  NBC  Reconnaissance 

FM  3-50  Smoke  Operations 

FM  3-100  NBC  Defense,  Chemical  Warfare,  Smoke,  and  Flame  Operations 
FM  3-101  Chemical  Staffs  and  Units 

FM  5-7-30  Brigade  Engineer  and  Engineer  Compare  Combat  Operations  (Airborne,  Air 
Assault,  Light) 

FM  5-33  Terrain  Analysis 
FM  5-34  Engineer  Field  Data 

FM  5-71-3  Brigade  Engineer  Combat  Operations  (Armored) 

FM  5-71-100  Division  Engineer  Combat  Operations 
FM  5-100  Engineer  Operations 
FM  5-100-15  Corps  Engineer  Operations 

FM  6-20-10  Tactics,  Techniques,  and  Procedures  for  the  Targeting  Process 
FM  6-20-20  Tactics,  Techniques,  and  Procedures  for  Fire  Support  at  Battalion  Task  Force 
and  Below 

FM  6-20-30  Tactics,  Techniques,  and  Procedures  for  Fire  Support  for  Division  and  Corps 
FM  6-20-40  Fire  Support  for  Brigade  Operations  (Heavy) 

FM  6-20-50  Tactics,  Techniques,  and  Procedures  for  Fire  Support  for  Brigade  Operations 
(Light) 

FM  6-60  Tactics,  Techniques,  Procedures  for  the  Multiple  Launch  Rocket  System  (MLRS) 
Operations 

FM  6-71  Tactics,  Techniques  and  Procedures  for  Fire  Support  for  the  Combined  Arms 
Commander 

FM  6-121  Tactics,  Techniques,  and  Procedures  for  Field  Artillery  Target  Acquisition 

FM  7-90  Tactical  Employment  of  Mortars 

FM  8-10  Health  Service  Support  in  a  Theater  of  Operations 

FM  9-43-2  Recovery  and  Battlefield  Damage  Assessment  and  Repair 

FM  10-23  Basic  Doctrine  for  Army  Field  Feeding 

FM  10-52  Water  Supply  in  the  Theater  of  Operations 

FM  11-32  Combat  Net  Radio 
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FM  11-37  MSE  Primer  for  Small-Unit  Leaders 
FM  1 1-43  The  Signal  Leaders  Guide 

FM  11-50  Combat  Communications  within  the  Division  (Heavy  and  Light) 

FM  12-6  Personnel  Doctrine 
FM  16-1  Religious  Support 

FM  19-40  Enemy  Prisoners  of  War,  Civilian  Internees,  and  Detained  Persons 

FM  20-32  Mine/Countermine  Operations 

FM  25-101  Battle  Focused  Training 

FM  33-1  Psychological  Operations 

FM  34-1  Intelligence  and  Electronic  Warfare  Operations 

FM  34-2  Collection  Management 

FM  34-2-1  Reconnaissance  and  Surveillance  and  Intelligence  Support  to 
Counterreconnaissance 
FM  34-3  Intelligence  Analysis 

FM  34-8  Combat  Commander ’s  Handbook  on  Intelligence 

FM  34-10  Division  Intelligence  and  Electronic  Warfare  Operations 

FM  34-10-2  Intelligence  and  Electronic  Warfare  Ecpiipment  Handbook 

FM  34-25  Corps  Intelligence  and  Electronic  Warfare 

FM  34-25-1  Joint  Surveillance  Target  Attack  Radar  System  (Joint  STARS) 

FM  34-25-3  All-Source  Analysis  System  and  the  Analysis  and  Control  Element 
FM  34-37  Echelons  Above  Corps  (EAC)  Intelligence  and  Electronic  Warfare  (lEW) 
Operations 

FM  34-52  Intelligence  Interrogation 
FM  34-60  Counterintelligence 

FM  34-80  Brigade  and  Battalion  Intelligence  and  Electronic  Warfare 

FM  34-81  Weather  Support  for  Army  Tactical  Operations 

FM  34-81-1  Battlefield  Weather  Effects 

FM  34-130  Intelligence  Preparation  of  the  Battlefield 

FM  41-10  Civil  Affairs 

FM  43-5  Unit  Maintenance  Operations 

FM  44-43  Bradley  Stinger  Fighting  Vehicle  Platoon  and  Squad  Operations 
FM  44-64  FAAD  Battalion  and  Battery  Operation 
FM  44-100  US  Army  Air  Defense  Operations 
FM  55-12  Movement  of  Units  in  Air  Force  Aircraft 
FM  55-15  Transportation  Reference  Data 

FM  63-2  Division  Support  Command,  Armored,  Infantry,  and  Mechanized  Infantry  Divisions 
FM  63-20  The  Forward  Support  Battalion 
FM  63-21  Main  Support  Battalion 

FM  71-2  The  Tank  and  Mechanized  Infantry  Battalion  Task  Force 
FM  71-3  The  Armored  and  Mechanized  Infantry  Brigade 
FM  71-100  Division  Operations 

FM  71-123  Tactics  and  Techniques  for  Combined  Arms  Heavy  Forces:  Armored  Brigade, 
Battalion  Task  Force,  and  Company  Team 
FM  90-4  Air  Assault  Operations 
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FM  90-13-1  Combined  Arms  Breaching  Operations 

FM  90-15  J-SEAD,  Multi-Service  Procedures  for  the  Joint  Suppression  of  Enemy  Air 
Defenses 

FM  90-21  JAAT,  Multi-Service  Procedures  for  the  Joint  Air  Attack  Team  Operations 

FM  90-25  ALCO,  Airlift  for  Combat  Operations 

FM  100-2-3  The  Soviet  Army  -  Troops,  Organization,  and  Equipment 

FM  100-5  Operations 

FM  100-9  Reconstitution 

FM  100-10  Combat  Service  Support 

FM  100-16  Army  Operational  Support 

FM  100-26  Air  Ground  Operations  System 

FM  100-42  US  Air  Force/US  Army  Airspace  Management  in  an  Area  of  Operations 
FM  100-103  Army  Airspace  Command  and  Control  in  a  Combat  Zone 
FM  100-103-1 ICAC2,  Multi-Service  Procedures  for  Integrated  Combat  Airspace  Command 
and  Control 

FM  100-103-2  TAGS,  Multi-Service  Procedures  for  the  Theater  Air  Ground  System 
FM  101-5  Staff  Organization  and  Operations 
FM  101-5-1  Operational  Terms  and  Symbols 

FM  101-10-1/2  Staff  Officers  Field  Manual  Organizational,  Technical,  and  Logistical  Data 
Planning  Factors  (Volume  2) 

Other  Publications 

AR  165-1  Chaplain  Activities  in  the  US.  Army 

AR  190-8  Enemy  Prisoners  of  War,  Administration,  Employment  and  Compensation 

AR  600-8  Military  Personnel  Management 

AR  600-8-6  Personnel  Accounting  and  Strength  Reporting 

Command  and  General  Staff  College  Student  Text  (ST)  101-5  Command  and  Staff  Decision 
Processes,  Feb  95 

US  Army  Air  Defense  Artillery  School  Student  Text  44-43-1  BFVS  Platoon  and  Squad 
Operation 

DA  Pamphlet  27-25  Prisoner  of  War:  Rights  and  Obligations  Under  the  Geneva  Convention 
TC  12-16  PAC  Noncommissioned  Officer's  Guide 
TC  12-17  Adjutant's  Call  the  S-J's  Handbook 

TC  16-2  Religious  Support  to  Casualties,  Memorial  and  Funeral  Services 
TM  1 1-5280-890-10-1  Technical  Manual  Operator 's  Guide  for  the  RT-1523 
TM  11-701 0-2 13-12  Operator 's  and  Organizational  Maintenance  Manual;  Tactical  Army 
Combat  Service  Support  Computer  System  AN/TYQ-33(V) 

ARTEP  1  \-2-MI?  Mission  Training  Plan  for  the  Tank  and  Mechanized  Infantry  Battalion 
Task  Force 

ARTEP  71-3  Mission  Training  Plan  for  the  Heavy  Brigade  Command  Group  and  Staff 
FC  71-6  Battalion  and  Brigade  Command  and  Control 

TRADOC  Pamphlet  350-16  Heavy  Opposing  Force  (OPFOR)  Tactical  Handbook 
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US  Army  Air  Defense  Artillery  School  Graphic  Training  Aid  (GTA)  AA-1-\Q  Aircraft 
Recognition  Playing  Cards 

US  Army  Chaplain  Center  and  School  Reference  Book,  RB  1-1  Unit  Ministry  Team  (UMT) 
Handbook 

Battle  Command  Battle  Laboratory  Leadership  and  Decision  Making  for  War  and 
Operations  Other  Than  War  (Draft  2.1)  Apr  94. 

Battle  Command  Battle  Laboratory  Command  Techniques  and  Procedures  (1st  Coord 
Draft)  Apr  95 

Army  Research  Institute,  Fort  Benning  Field  Unit  Commander’s  Battle  Staff  Handbook  with 
Garrison  Duties,  15  May  1993 

Army  Research  Institute  Fort  Benning  Field  Unit  Brigade  Commander's  Battle  Staff 
Handbook,  11  March  1995 

Center  for  Army  Lessons  Learned  (CALL)  Newsletter  91-1  Rehearsals,  April  1991 
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APPENDIX  B 


Formative  Evaluation  Forms 

ITTBBST-BSTS  materials  were  evaluated  through  the  use  of  military  subject  matter 
experts  conducting  alpha  and  beta  testing  of  text  and  computer-based  instruction.  In  addition  to 
student  evaluative  comments  of  the  material,  demographic  and  doctrinal  data  was  collected 
through  the  use  of  multi-page  forms. 

The  primary  focus  of  Alpha  test  participants  was  the  doctrinal  accuracy  of  materials. 

Alpha  testers  provided  written  comments  on  the  doctrinal  accuracy  of  text  and  CBI  materials,  as 
well  as  the  value  of  practical  exercises  and  exams  in  tr^ning  the  material  and  evaluating 
knowledge. 

Beta  testers  were  instructed  to  take  the  course  as  though  they  were  regular  students. 
Participants  completed  questionnaires,  participated  in  individual  interviews,  and  provided  on-line 
evaluation  and  feedback. 

Student  packets  of  Formative  Evaluation  (FE)  forms  could  be  divided  into  three  main 
categories:  demographic  data,  comments  on  the  instruction/evaluations,  and  comments  on  the 
value  of  the  various  course  components.  Examples  of  a  form  in  each  category  are  included  in  this 
appendix. 

While  student  comments  were  vital  to  the  course  development/evaluation  process, 
demographic  data  enabled  comments  to  be  placed  in  proper  perspective.  The  lessons  learned 
from  BSTS  (main  body  of  this  report)  were  developed  through  a  study  of  student  comments, 
cross-referenced  with  demographic  data  to  give  a  reasonable  understanding  of  the  framework  in 
which  the  remarks  should  be  placed. 

Students  provided  data  on  age,  educational  level,  computer  experience,  service  component 
(AC,  ARNG,  USAR,  IRR),  military  qualifications,  and  assignments  relevant  to  the  course  being 
evaluated.  Demographic  data  provided  as  complete  a  picture  as  possible  of  the  individual  doing 
the  Alpha  or  Beta  testing. 

Comments  on  the  instructional  material  and  course  components  were  incorporated  into 
reports  to  the  COR  (extract  shown  in  appendix  D). 
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(NOTE:  This  is  the  first  page  of  an  eight  page  form  for  collecting  demographic  data.) 
ITTBBST  FORMATIVE  EVALUATION  FORM 

Last  name _ First  Name _ MI_  Date _ 

Local  Tel  # _ 

Unit  address _ .Unit  Tel  # _ 


Check  the  year  of  your  birth 

□78  □??  076  075  074  073  072  071  070  ^69  068  067 

□66  ^65  ^64  ^63  ^62  ^61  ^60  ^59  ^58  □57  ^56  □SS 

□54  ^53  ^52  ^51  ^50  ^49  ^48  ^47  ^46  ^45  ^44  ^43 

Check  highest  educational  level  completed 

□GED  ^12  □n  □H  □15  ^16  □n  ^18  ^19  ^20  ^21  ^22 


Check  baccalaureate 

riBiochemistry 
ncivil  Engineering 
□Educ.  Psychology 
□Foreign  Language  Ed. 
□Human  Relations 
□Justice  Administration 
□Training/Development 
□Secondary  Education 
□other  (specify) _ 


degree(s)  you  have  earned 

□Biology  □Business  Admin. 

□Classical  Languages  □Computer  Science 
□Electrical  Engineering  □English 
□Geography/Geoscience  □Health/Physical  Ed. 
□Humanities 
□Linguistics 


□Philosophy 

□Sociology 


□chemistry 

□Early/Middle  Education 
□Fine  Arts 

_  ,  □History 

□industrial  Engineering  □information  Science 
□Mathematics  □Mechanical  Engineering 

□Physiology/Biophysics  □Public  Administration 
□Systems  Science/ORSA^Uiban/Public  Affairs 
_ □No  baccalaureate  degrees  earned 


Check  advanced  degree(s)  you  have  earned 

□Biochemistry  □Biology  □Bitsiness  Admin.  □Chemistry 

□civil  Engineering  □Classi^  Languages  □Computer  Science  □Early/Middle  Education 

□Educ.  Psychology  □Electrical  Engineering  □English  □Fine  Arts 

□Human  Relations  □Humanities  □industrial  Engineering  □information  Science 

□Justice  Administration  □Linguistics  □Mathematics  □Mechanical  Engineering 

□Training/Development  □Philosophy  □Physiology/Biophysics  □Public  Administration 

r~l Secondary  Education  □Sociology  □Systems  Science/ORSA^Urban/Public  Affairs 

□other  (specify) _ □No  advanced  degrees  earned 


Check  your  current  component 
□Army  (active)  □ARNG  □USAR 
Check  year  you  entered  basic  active  service 
□96  ^95  ^94  ^93  ^92  ^91 

□84  □SJ  ^82  ^81  ^80  ^79 

□72  ^71  ^70  ^69  ^68  ^67 


□iRR  (includes  retired  persormel) 


□90 

□78 

□66 


□89 

□77 

□65 


□88 

□76 

□64 


If  ARNG  or  IISAR,  check  year  you  left  active  service  (do  not  include  drills) 


□96  ^95  ^94  ^93  ^92  ^91  ^90  ^89  ^88 

□84  ^83  ^82  ^81  ^80  ^79  ^78  ^77  ^76 

If  retired  or  separated^  check  year  of  retirement  or  separation 
□96  ^95  ^94  ^93  ^92  ^91  ^90  ^89  ^88 

Check  current  rank  or  last  rank  held  before  separation  or  retirement 

□gen  □ltg  □mg  □bg  □col  □ltc  □maj  □CPT  □ILT 

□CW5  □CW4  □CW3  □CW2  □WOl 


□87 

□75 

□63 

□87 

□75 

□87 

□2LT 


□86 

□85 

□74 

□73 

□62 

□61 

□86 

□85 

□74 

□73 

□86 

□85 
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(NOTE:  This  form  used  to  evaluate  pretest/post  test  for  each  subject.) 


Doctrinal  content  of  this  subject  Pretest/Posttest.  Check  the  block  that  indicates 
your  opinion  for  each  question  listed.  A  rating  of  1  or  2  requires  comments. 

1.  Tests  no  useful  knowledge/would  never  ask  this. 

2.  Tests  minimal  knowledge/would  rarely  ask  this. 

3.  Tests  good  basic  knowledge/might  well  ask  this. 

4.  Tests  advanced  materials/would  often  ask  this. 

5.  Tests  complete  and  thorough  knowledge/would  always  ask  this. 

Record  your  pretest  and  posttest  scores  for  both  the  subject  pretest  and  posttest  as 
well  as  how  long  each  test  took.  Place  comments  in  the  block  to  the  right  of  each 
item. 


Posttest:  Score _  Start 

Stop 


□□□□□  2._ 
□□□□□  3“ 

□□□□□  4~ 

□□□□□  iZ 
□□□□□  6” 
□□□□□  T 
□□□□□  iZ 
□□□□□ 
□□□□□  10. 


Pretest:  Score_^ 

Time _ 

1. 2. 3. 4. 5. 

Time _ 

□□□□□  1. 


Start  Time 


Stop  Time_ 
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(NOTE:  This  form  used  to  evaluate  lesson  material;  tailored  for  each  lesson  of  each  course.) 

Doctrinal  content  of  text  and  CBL  Check  the  block  that  indicates  your  opinion  for  each  area  listed 
(i.e.  how  closely  the  material  matches  approved  doctrine  such  as  FMs,  ARTEPs,  and  ARs  -  there  is 
a  list  of  these  references  in  your  materials). 

1.  No  useful  knowledge/would  never  use/would  need  more  information  for  every  situation. 

2.  Minimal  knowledge/would  rarely  use/would  probably  need  more  information  for  basic 
situations. 

3.  Good  basic  knowledge/could  use  for  basic  situations/may  require  more  information  for  complex 
situations. 

4.  Advanced  materials/suitable  for  entry  level/standard  use  for  normal  function  of  job  incumbent 

5.  Complete  and  thorough  knowledge/could  use  for  all  situations/would  require  no  assistance  for 
any  situations. 

Record  start  time  and  end  time  for  each  lesson.  Remember  that  you  will  time  how  long  it  took  to 
complete  CBI  and  text  materials  separately.  Place  comments  in  the  block  to  the  right  of  each  item 
and  continue  (if  necessary)  to  the  back  of  the  page.  Be  sure  to  include  the  subject,  lesson  and  topic 
number  -  i.e.  lA-1.  When  commenting  on  a  lesson  screen,  use  the  number  shown  in  the  upper  left 
comer  of  the  screen  to  identify  your  comment  A  rating  of  1  or  2  requires  comments. 

Subject  One  -  The  Battalion  Battlefield 

Time _ 

Lesson  lA  -  Army  Operations  Doctrine 

Lesson  Exam  Score: _ Take  from  next  page  after 

1.  2.  3.  4.  5.  taking  the  lesson  exam! 

□□□□□  lA  Objective _ 

□□□□□  lA  Introduction _ 

□□□□□  lA  1.  Structure  of  Modem  Battlefield _ 

□□□□□  1A2.  Principles  of  War _ 

□□□□□  1A3.  Combat  Power _ 

□□□□□  1A4.  Tenets  of  Army  Operations _ 

□□□□□  1 A  5.  Maneuver  Battalion  Task  Force  Mission 

□□□□□  1 A  Conclusion _ 


Lesson  Comments 

Start  Time _ Stop 

Reference  read  min.  used: _ 

TEXT  lesson  minutes  used:_ 
CBI  lesson  minutes  used:  ^ 
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(NOTE:  This  form  used  to  evaluate  lesson  exam  for  each  lesson.) 

Doctrinal  content  of  this  lesson  Exam.  Check  the  block  that  indicates  your  opinion  for  each  question  listed. 
A  rating  of  1  or  2  requires  comments. 

1.  Tests  no  useful  knowledge/would  never  ask  this. 

2.  Tests  minimal  knowledge/would  rarely  ask  this. 

3.  Tests  good  basic  knowledge/might  well  ask  this. 

4.  Tests  advanced  materials/would  often  ask  this. 

5.  Tests  complete  and  thorough  knowledge/would  always  ask  this. 

Record  your  score  for  the  lesson  exam  as  well  as  how  long  the  test  took.  Place  comments  in  the  block  to  the 
right  of  each  item. 

Score:  Exam _ Start  Time _ Stop  Time _ 

1.  2.  3.  4.  5. 

□□□□□  1 _ _ 


□□□□□  2.. 


□□□□□  3. 


□□□□□  4. 


□□□□□  5. 


□□□□□  6. 


□□□□□  7. 


□□□□□  8.. 


□□□□□  9. 


□□□□□  10. 
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(NOTE:  This  form  used  to  evaluate  value  of  course  components.) 


Please  answer  the  following  questions  using  the  five  point  assessment  scale  shown  helow.  Provide  additional 
comments  in  the  spaces  provided.  Questionnaires  are  identified  by  number  only,  so  please  be  candid. 


1.  Strongly  disagree. 

2.  Disagree. 

3.  Undecided. 

4.  Agree. 

5.  Strongly  agree. 


1.  2.  3. 4.  5. 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 

□□□□□ 


1.  The  doctrinal  manuals  provided  were  current. 

2.  The  reading  assignments  were  accurate  as  to  chapter,  aimex,  and  so  on  in  the  manuals. 

3.  The  reading  assignments  were  appropriate. 

4.  Scenario  based  questions  made  the  test  more  challenging. 

5.  The  computer  log-in  procedures  were  easy  to  use. 

6.  The  BSTS  menus  and  controls  (buttons,  sliders,  and  mouse  clicks)  worked. 

7.  I  was  able  to  control  which  lessons  I  took  easily. 

8.  The  text  portions  of  the  course  meshed  well  with  the  CBI  portions. 

9.  Navigation  through  the  CBI  portion  of  the  course  was  easy. 

10.  Audio  quality  was  sufficient  to  clearly  understand  the  narration. 

11.  The  workbook  was  easy  to  use. 

12.  I  did  not  have  to  wait  for  the  computer  to  refresh  the  screen. 

13.  The  colors  used  for  the  on-screen  text  during  the  CBI  lessons  were  easy  to  see. 

14.  I  had  enough  previous  computer  experience  to  complete  the  course  when  I  arrived. 

15.  The  video  CBI  instruction  was  clear. 

16.  The  video  CBI  instmction  played  without  dropping  video  frames  (jerking  picture). 

17.  The  video  CBI  instruction  played  without  dropping  audio  frames  (jerking  sotmd). 

18.  The  videos  were  appropriate  to  the  lessons. 

19.  The  pace  of  the  course  was  right. 

20.  The  level  of  the  instruction  was  appropriate. 


Please  rank  order  the  following  course  components  as  to  their  educational/training  value,  with  five  being  the 
most  valuable.  (Assign  only  one  value  to  each.) 


_ Work  book _ Job  Aids _ References  (FMs,  etc.) _ CBI _ Exercises 

What  suggestions  do  you  have  for  improving  this  instruction  or  method  of  presentation?  Use  additional  sheets  if 
necessary. 
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(NOTE;  This  form  used  to  evaluate  COMPS.) 

Doctrinal  content  of  this  comprehensive  exam.  Check  the  block  that  indicates  your  opinion  for  each 
listed.  A  rating  of  1  or  2  requires  comments. 

1.  Tests  no  useful  knowledge/would  never  ask  this. 

2.  Tests  minimal  knowledge/would  rarely  ask  this. 

3.  Tests  good  basic  knowledge/might  well  ask  this. 

4.  Tests  advanced  materials/would  often  ask  this. 

5.  Tests  complete  and  thorough  knowledge/would  always  ask  this. 

1.2.  3.  4.  5.  QUINT# 

□□□□□ _ 19 _ _ 


□□□□□ _ 20. 


□□□□□ _ 21 . 


□□□□□ _ 22. 


□□□□□ _ 23. 


□□□□□ _ 24. 


□□□□□ _ 25. 


□□□□□ _ 26. 


□□□□□ _ 27. 


□□□□□ _ 28. 


□□□□□ _ 29. 


□□□□□ _ 30. 
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APPENDIX  C 


BSTS  Target  Audience  Description 

The  training  audience,  identified  in  the  fi'ont-end  analysis,  consisted  of  13  positions  at 
each  echelon.  At  battalion  level  there  is  a  Chaplain,  but  no  S5.  At  brigade,  the  Chaplain  is 
combined  with  the  SI  and  the  S5  is  added.  Note  that  not  all  brigades  will  have  an  S5 
assigned;  however,  due  to  the  increasing  importance  of  Civil-Military  Operations,  the  S5 
course  is  included. 


BN  -  BSTS  Target  Audience 

Battalion  Executive  Officer  -  A  Combat  Arms  (Infantry  or  Armor)  Major  or  promotable 
Captain;  having  served  as  company  commander,  and  a  graduate  of  both  the  CAS  and  CGSC. 

Adjutant  (SI)  -  A  Combat  Arms  Captain,  branch  qualified,  and  a  graduate  of  OAC. 

Intelligence  Officer  (S2)  -  A  Military  Intelligence  Captain,  branch  qualified  and  a  graduate  of 
OAC. 

Operations  Officer  (S3)  -  A  Combat  Arms  Major,  having  commanded  a  company,  branch 
qualified,  and  a  graduate  of  CAS^  and  CGSC. 

S3  Air  Officer  (S3  A)  -  A  Combat  Arms  Captain,  branch  qualified,  having  completed  OAC. 

Logistics  Officer  (S4)  -  A  Combat  Arms  Captain,  branch  qualified  and  having  completed 
OAC. 

Battalion  Chaplain  -  An  Army  Chaplain,  branch  qualified. 

Signal  Officer  -  A  Captain,  Signal  Corps,  branch  qualified,  and  having  completed  OAC. 

Fire  Support  Officer  -  A  Field  Artillery  Captain,  branch  qualified,  and  having  completed  OAC. 

Engineer  Officer  -  An  Engineer  Captain,  branch  qualified,  and  having  completed  OAC. 

Chemical  Officer  -  A  LT,  Chemical  Corps,  branch  qualified,  and  having  completed  OBC. 

Air  Defense  Artillery  Officer  -  An  Air  Defense  Artillery  LT,  branch  qualified,  and  having 
completed  OBC. 
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BDE  -  BSTS  Target  Audience 


Brigade  Executive  Officer  -  A  Combat  Arms  (Infantry  or  Armor)  Lieutenant  Colonel,  branch 
qualified,  and  a  graduate  of  CAS^  and  CGSC. 

Adjutant  (SI)  -  A  Combat  Arms  Major,  branch  qualified,  and  a  graduate  of  OAC  and  CAS^. 

Intelligence  Officer  (S2)  -  A  Military  Intelligence  Major,  branch  qualified  and  a  graduate  of 
OAC  and  CAS^ 

Operations  Officer  (S3)  -  A  Combat  Arms  Major,  branch  qualified,  and  a  graduate  of  CGSC. 

S3  Air  Officer  (S3 A)  -  An  Infantry  Captain,  branch  qualified,  having  completed  CAS^. 

Logistics  Officer  (S4)  -  A  Quartermaster  Corps  Major,  branch  qualified  and  having  completed 
CAS^ 

Civil  Affairs  Officer  (S5)  -  A  Combat  Arms  Captain  or  Major,  branch  qualified,  and  having 
completed  CAS^.  (Note:  The  S5  is,  normally,  not  aurthorized  by  TOE  or  MTOE  except  in 
the  Ranger  Regiment.  The  S5  is  an  augmentee  to  the  existing  MTOE.) 

Signal  Officer  -  A  Signal  Corps  Captain,  branch  qualified,  and  having  completed  CAS^. 

Fire  Support  Officer  -  A  Field  Artillery  Major,  having  commanded  a  battery,  branch  qualified, 
and  having  completed  CAS^. 

Engineer  Officer  -  An  Engineer  Corps  Major,  branch  qualified  and  having  completed  CAS^. 

Chemical  Officer  -  A  Chemical  Corps  Captain,  branch  qualified  and  having  completed  CAS^. 

Air  Defense  Artillery  Officer  -  An  Air  Defense  Artillery  LT,  branch  qualified,  and  having 
completed  OBC. 
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APPENDIX  D 

Phase  III  -  Beta  -  Sample  Report  Extract 


January  01,  1997 


(706)  682-4699 
BDM/FTB-CRA-0504-97 


U.S.  Army  Research  Institute 

Armored  Forces  Research  Unit 

ATTN:  PERI-IK  (Dr.  Kathleen  A.  Quinkert) 

Ft.  Knox,  Kentucl^  40121-5620 

Dear  Dr.  Quinkert: 

The  purpose  of  this  letter  is  to  provide  you  the  Final  results  of  the  BETA  Test 
conducted  15-30  May  1996  at  Ft.  Knox,  Kentucky  on  the  Brigade  S5  Officer’s  course. 
This  test  was  completed  in  compliance  wth  the  approved  design  document  for  contract 
MDA903-92-D-0075,  delivery  order  0041 .  The  complete  final  beta  test  results  are 
attached  at  the  enclosure. 

By  way  of  an  executive  summary,  the  conclusions  and  recommendations 
derived  from  this  test  are  as  follows: 

Conclusions: 


The  course  of  instruction  was  tested  by  one  student  who  was 
representative  of  the  target  population,  and  one  student  who  had  no  staff 
experience  at  either  brigade  or  battalion.  The  tester  who  had  had  battalion  and 
brigade  staff  experience  was  a  ci\dl  affairs  officer,  however  he  also  had  ALPHA 
tested  this  course  earlier.  The  time  spent  in  CBI,  as  recorded  by  this  tester,  is  not 
an  accurate  assessment  of  the  time  it  takes  the  student  to  do  the  CBI.  (There  were 
three  lessons  for  which  zero  time  was  logged  for  CBI;  however,  each  of  these 
three  lesson  examinations  are  conducted  in  CBI.)  These  circumstances  diminish 
the  value  of  the  data  collected. 

The  course  of  instruction  meets  the  terminal  learning  objectives  as 
measured  by  the  percentage  increase  in  examination  scores  from  the  pretest  to 
posttest. 
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Dr.  Kathleen  A.  Quinkert 
BDM/FTB-CRA-00504-97 
January  01,  1997 
Page  2 


The  course  of  instruction  meets  the  lesson  training  objectives  as  measured 
by  the  lesson  examination  scores  meeting  or  exceeding  the  benchmark  score  in  all 
but  lesson  1  A.  In  the  case  of  lesson  1  A,  the  civil  affairs  officer  met  the  benchmark 
score,  whereas  the  Captain  of  infantry  did  not  meet  the  benchmark  score. 

Recommendations: 


That  no  further  BETA  testing  be  conducted. 

That  research,  follow-up,  and  revisions  to  the  course  of  instruction  be 
accomplished  as  stated  at  enclosure  3. 

That  the  final  text  be  prepared  and  the  final  CD  be  cut. 

That  future  BETA  test  participants  be  screened  more  carefully  to  ensure 
they  possess  the  prerequisite  experience  and  MOS  base  to  conduct  a  valid  BETA 
test  of  the  course. 

That  care  be  taken  to  disallow  an  officer  who  has  acted  as  the  ALPHA 
tester  for  a  course,  so  ensuring  they  did  not  BETA  test  the  same  course. 

The  above  recommendations  have  been  executed  and  are  included  in  the  final  training 
support  package  that  will  be  delivered  for  your  approval. 

Sincerely, 


Charles  R.  Andre' 

Director,  Training  and  Information  Systems 

1  Enclosure 

cc:  Dr.  B.  Black,  ARI 

T.  Lewman,  BDM/Monterey 
R.  Sever,  BDM/Ft.  Knox 

CRA/dmn 
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BATTLE  STAFF  TRAINING  SYSTEM 
FINAL  TESTING  SLMMARY  AND  RECOMMENDATIONS 
BRIGADE  S5  COURSE  1  JANUARY  1997 


I.  Target  Population  Demographics:  This  course  is  designed  to  prepare  the  newly  assigned 
brigade  S5  and  for  civil  military  operations  at  the  brigade  level.  It  is  designed  to  train  the  S5  in 
matters  concerning  the  civilian  impact  on  military  operations,  and  the  political,  economic,  and 
social  effects  of  military  operations  on  a  civilian  population.  Ordinarily,  the  brigade  S5  will  be  a 
senior  (promotable)  Captain,  or  a  Major.  He  should  possess  excellent  staff  cooperating  and 
coordinating  skills,  and  be  able  to  work  with  host  nation  governments  and  special  authorities. 

A.  Requested.  The  officers  selected  to  BETA  test  this  course  should  be  senior  Captains 
(promotable)  or  Majors.  They  should  be  branch  qualified  in  one  of  the  combat  arms,  graduates 
of  basic  and  advance  courses,  and  should  have  attended  the  combined  arms  and  services  staff 
school.  They  should  have  experience  as  platoon  leaders,  company  commanders,  and  battalion 
or  brigade  staff  officers. 

B.  Actual.  The  two  officers  selected  to  BETA  test  this  course  differed  widely  in 
demographics.  One  oflBcer  was  a  Captain  of  Infantry,  age  36,  with  ten  years  active  duty  service, 
and  a  graduate  of  The  Combined  Arms  and  Services  Staff  School.  All  his  experience  had  been 
at  company  level.  He  has  participated  in  three  rotations  at  the  national  training  center,  and  has 
no  combat  experience.  The  other  officer,  a  Lieutenant  Colonel  of  Civil  Affairs,  age  44,  with 
twenty-one  years  in  the  service,  is  now  a  civil  affairs  officer,  branch  trained,  who  has  attended 
engineer,  military  police,  civil  affairs,  and  special  operations  forces  advance  courses,  and  the 
resident  command  and  general  staff  officer  course.  His  combat  experience  was  in  Bosnia,  and 
he  has  no  combat  training  center  experience.  He  has  served  as  special  operations  forces  cadre 
at  the  joint  readiness  training  center. 

C.  Target  Population  Summary.  One  of  the  two  oflBcers  was  representative  of  the  target 
population.  The  Lieutenant  Colonel,  Civil  Affairs,  is  the  same  officer  who  Alpha  tested  this 
course.  Because  of  the  pre-test  (Alpha)  bias  of  one  of  the  testers,  and  the  lack  of  experience  of 
the  other,  the  resulting  data  are  considered  inadequate  and  partially  contaminated.  These 
circumstances  diminish  the  value  of  the  test  data. 

n.  Instructional  Results.  (Note:  The  following  times  and  scores  are  averages  from  the 
population  sample.) 

A.  The  course  consists  of  four  basic  actiwties;  diagnostic  testing,  reading  (study  materials 
and  text  lessons),  computer-based  instruction,  and  achievement  testing.  Each  of  the  BETA  test 
students  was  asked  to  record  his  scores  on  all  examinations,  and  to  keep  track  of  the  actual  time 
spent  in  negotiating  the  examinations,  text  lessons  and  computer  based  instruction.  The  results 
of  these  four  activities  were  analyzed,  and  the  results  are  as  follows: 
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B.  Pre/post  test:  The  subject  tests  were  used  as  both  diagnostic  and  achievement  tests. 
Scores  on  the  pretests  were  compared  with  those  on  the  posttests  to  determine  the  differences 
between  the  two  scores,  and  to  indicate  whether  or  not  the  students  had  met  the  standard.  The 
results  of  this  comparison  are  shown  in  Table  1. 


PRE/POST  1 

[EST  COMPARISON  (N=2) 

SUBJECT 

PRETEST 

FINAL 

EXAM 

%  DIFFERENCE 

STANDARD 

MET? 

1.  Civil  Affair 

Operations 

75% 

94% 

25.50% 

YES 

2.The  Brigade  S5 

72% 

79% 

9.72% 

NO 

Table  1 


C.  Lesson  Examinations:  Results  of  lesson  examinations  are  displayed  at  Table  2. 


LESSON  EXAM  SUMMARY  (N=2) 


LESSON 

SCORE 

STANDARD  MET? 

1 A  Civil  Affairs  Overview 

73.00 

NO 

IB  Psychological  Operations 
Overview 

92.00 

YES 

1C  Enemy  Prisoners  of  War  & 
Civilians 

80.00 

YES 

2  A  The  Brigade  S5 
Responsibilities 

90.00 

YES 

2B  Staff  Integration 

80.00 

YES 

2C  Establishing  a  Dislocated 
Civ  Ctr 

100.00 

YES 

Table  2 


D.  Comprehensive  Examination.  The  average  score  on  the  end  of  course  comprehensive 
examination  was  70.50.  The  officer  who  made  the  lower  score  reported  study  time  of  eleven 
hours  for  text,  and  four  hours  for  doctrine.  His  study  time  notwithstanding,  his  lack  of 
experience  may  have  placed  him  at  a  disadvantage  in  this  case. 

E.  BETA  Test  Comments. 

1.  Subjective  Comments.  Subjective  comments  are  at  Enclosure  3 

2.  Structured  Post-Course  Comments.  Structured  comments  indicate  that  the  course 
is  a  good,  solid  entry  level  course.  That  is  to  say,  that  after  the  student  finishes  this  course,  he 
or  she  could  go  to  a  brigade  and  fulfill  the  duties  of  the  brigade  S5. 
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F.  Course  Length  Summary: 

1.  Time  to  Complete: 

TEXT:  13.67 

DOCTRINE:  2.83 

CBI:  4.28 

20.78  Hours 

2.  Analysis  of  student-reported  times. 

(a)  Text  Lesson  Reading  Times.  The  Course  Time  Analysis,  shows  that  the 
students  did  not  read  the  full  text.  Indications  are  that  they  went  right  to  the  examination  and 
“gamed”  their  way  through  the  course,  instead  of  attempting  to  accomplish  the  performance 
objectives  set  forth  in  the  course  front  matter.  This  may  account,  in  part,  for  the  relatively  low 
average  score  on  the  end  of  course  comprehensive  test. 

(b)  Doctrinal  Publication  Reading  Times.  Of  a  total  of  approximately  42 
hours  (calculated  at  12  pages  per  hour)  of  doctrinal  reference  reading  requirements,  the  students 
averaged  slightly  less  than  seven  hours,  or  approximately  17%.  This  number  represents  unique 
reading  requirements  (if  a  reference  was  required  by  lesson  1 A  and  IB,  for  example,  the  number 
of  pages  was  included  only  once  in  the  total  page  count.)  This  is  becoming  a  trend  -  students 
either  cannot  or  will  not  read  the  doctrinal  requirements. 

(c)  Computer-Based  Instruction.  The  average  time  spent  in  computer-based 
instruction  was  4.28  hours.  This  includes  all  lessons,  pretests  and  posttests,  and  the  end  of 
course  comprehensive  test.  The  analysis  of  CBI  time  indicates  little  or  no  time  spent  in  CBI  by 
tester  number  two.  Either  there  are  some  inaccuracies  in  his  own  record  keeping,  or  he  did  not 
do  the  examinations  for  lessons  lb,  Ic,  and  2b.  Zero  time  is  recorded  for  these  lessons.  In 
lessons  2A  and  2C,  both  of  which  contain  CBI  instructional  material,  he  recorded  five  minutes 
and  twenty  minutes  respectively.  The  CBI  data  for  tester  number  two  was  ignored  in 
calculating  the  CBI  time  for  the  course. 

m.  Conclusions  and  Recommendations. 

A.  Conclusions: 

1.  The  course  of  instruction  was  tested  by  one  student  who  was  representative  of  the 
target  population,  and  one  student  who  had  no  staff  experience  at  either  brigade  or  battalion. 
The  tester  who  had  had  battalion  and  brigade  staff  experience  was  a  civil  affairs  officer, 
however  he  also  had  ALPHA  tested  this  course  earlier.  The  time  spent  in  CBI,  as  recorded  by 
this  tester,  is  not  an  accurate  assessment  of  the  time  it  takes  the  student  to  do  the  CBI.  (There 
were  three  lessons  for  which  zero  time  was  logged  for  CBI;  however,  each  of  these  three  lesson 
examinations  are  conducted  in  CBI.)  These  circumstances  diminish  the  value  of  the  data 
collected. 
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2.  The  course  of  instruction  meets  the  terminal  learning  objective  as  measured  by  the 
percentage  increase  in  exam  scores  from  the  pretest  to  posttest. 

3.  The  course  of  instruction  meets  the  lesson  training  objectives  as  measured  by  the 
lesson  examination  scores  meeting  or  exceeding  the  benchmark  score  in  all  but  lesson  1  A.  In 
the  case  of  lesson  1  A,  the  civil  affairs  officer  met  the  benchmark  score,  whereas  the  Captain  of 
infantry  did  not  meet  the  benchmark  score. 

B.  Recommendations: 

1 .  That  the  COR  approve  the  exclusion  of  the  CBI  data  from  tester  number  two,  and 
that  only  the  CBI  data  from  tester  number  one  be  used  for  the  purpose  of  determining  the  length 
of  time  required  to  accomplish  the  CBI  in  the  course. 

2.  That  no  further  BETA  testing  be  conducted. 

3.  That  research,  follow-up,  and  revisions  to  the  course  of  instruction  be 
accomplished  as  stated  at  Enclosure  3. 

4.  That  the  final  text  be  prepared  and  the  final  CD  be  cut. 

5.  That  future  BETA  test  participants  be  screened  more  carefully  to  ensure  they 
possess  the  prerequisite  experience  and  MOS  base  to  conduct  a  valid  BETA  test  of  the  course. 

6.  That  care  be  taken  to  disallow  an  officer  who  has  acted  as  the  ALPHA  tester  for  a 
course  to  BETA  test  the  same  course. 
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Enclosure  1,  Post  Course  Structured  Comments 

The  Beta  Test  packets  contain  a  rating  sheet  for  each  lesson.  Each  topic  in  the  lesson  is 
rated  on  a  scale  of  one  to  five,  with  five  being  the  highest  value  assigned  to  the  material  under 
test.  The  topics  are  rated  as  to  their  value  in  enabling  the  student  to  perform  the  duties  of  the 
brigade  S5  as  they  pertain  to  that  particular  subject  and  lesson. 

Score  1  -  No  useful  knowledge.  Would  need  more  information  for  every  situation 

encountered  on  the  job. 


Score  2  -  Minimal  knowledge/would  rarely  use/would  probably  need  more  information  for 
basic  situations. 


Score  3  -  Good  basic  knowledge/could  use  for  basic  situations/may  require  more  information 
for  complex  situations. 


Score  4  -  Advanced  materials/suitable  for  entry  level/standard  use  for  normal  function  of  job 
incumbent.  _ _ _ 


Score  5  -  Complete  and  thorough  knowledge/could  use  for  all  situations/would  require  no 
assistance  for  any  situation. 


D-7 


Enclosure  2,  Subject  and  Lesson  Summary  for  Brigade  S5  Officer  Beta  Test 

Subject  1  -  Civil  Affair  Operations 

TIME:  (Hours) 

TEXT:  4.04 

DOCTRINE:  2.67 

CBI:  L90 

8.61  Hours 

Examination  Scores: 

PRETEST:  75% 

POSTTEST:  94% 

LESSONEXAM  lA:  73.00% 

LESSON  EXAM  IB:  92.00% 

LESSONEXAM  1C:  80.00% 

Subject  2  -  The  Brigade  SS 

TIME:  (Hours) 

TEXT:  2.79 

DOCTRINE:  0.17 

CBI:  2J5 

5.11  Hours 

Examination  Scores: 

PRETEST:  72% 

POSTTEST:  79% 

LESSON  EXAM  2A:  90.00% 

LESSON  EXAM  2B:  80.00% 

LESSON  EXAM  2C:  100.00% 


D-8 


Enclosure  3,  Brigade  S5  Officer  Beta  Tester  Formative  Evaluation 

[NOTE:  C  =  Checkyresearch  the  state  doctrinal/ 
grammatical  error  and  either  fix  or, 
if  not  required,  report  detailed  rational 
to  PM  with  documentation. 

F  =  Fix 

C/F  =  Check  then  Fix 
?  =  Unsure  of  issue,  research 
N/A  =  No  action  required] 

SUBJECT  ONE  -  CIVIL  AFFAIRS  OPERATIONS 
Lesson  1  Pre-Test 

Question  #  10  -  Sentence  missing  “range”  from  “short-range”  word. 

Response:  Question  corrected. 

Question  #19  -  The  last  sentence  gives  the  answer  away.  Also  “awide”  should  be  two  words. 
Response:  Question  rewritten  for  clarity 

Lesson  1 A  -  Civil  Affairs  Overview 

When  I  finished  with  Lesson  1 A  the  book  did  not  tell  me  to  go  to  the  computer. 

Response:  No  change.  All  lessons  are  standardized.  Student  failed  to  follow  instructions  from 
Student  Text. 

Course  intro,  has  graphic  image  only. 

Response:  Narration  added. 

Intro,  subject  graphic  says  “Civil  Affair”.  Needs  to  read  “Civil  Affairs”. 

Response:  Slide  replaced. 

Audio  on  intro,  uses  “Civil  Affair”.  Needs  to  say  “Civil  Affairs”. 

Response:  Narration  corrected. 

Introduction  Para  2  A  First  paragraph  include  in  CA  mission,  minimize  civilian  interference  with 
military  operations. 

Response:  Corrected.  Included  verbiage  to  definition. 


PRACTICAL  EXERCISE 
S5PE2C-18  Wrong  symbol. 

Response:  No  change.  Could  not  replicate  exact  graphic  lAW  FM  101-5-1. 
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S5  PE2C-25  No  replay  button. 

Response:  Fixed. 

On  exercise  170  is  not  necessarily  correct. 

UNACCOM  Males  158  =  23  tents 
UNACCOM  Females  35  =  5  tents 
UNACCOM  CIMCD  20  =  3+  tents 
Family  987  =  141+  tents 
mess,  process,  med  6  tents  =178  tents  total. 

Plus  10%  to  insure  families  are  not  overcrowded  or  split  up  10  tents.  Simple  book  calculations 
are  below  minimum.  Realistic  188  tents  minimum. 

Response:  No  change.  Student  was  asked  to  identify  minimum  requirement  for  billeting  only. 
170  is  the  correct  answer. 

One  iteration  of  problem  states  158  males  next  150  males,  changes  requirement  by  2  tents. 
Response:  Typographical  error  corrected. 

On  barbed  wire  calculations  you  determine  the  number  of  300  meter  3  sections  you  are  dealing 
with  and  then  how  much  wire  do  not  interpolate  1500  meters  =  9,  300  meter  sections.  9  times 
6  is  54  reels  unless  wood  pickets  are  used  then  63  reels. 

Response:  No  change.  Question  formula  is  correct  as  stated  lAW  FM  101-10-1/2. 

As  a  point  of  information  both  the  UNITCR  and  the  IRR  have  handbooks  on  camp 
development,  construction,  and  operation.  It  would  be  best  to  use  them  since  they  will  most 
likely  be  present,  you  will  be  getting  resources  from  them,  and  you  will  want  their  assistance  as 
much  as  possible.  So  do  it  their  way.  They  have  also  probably  done  it  before,  you  probably 
haven’t. 

Response:  No  change.  Recent  examples  of  operations  other  than  war  (OOTW),  notably 
Operation  Provide  Comfort,  have  indicated  that  combat  brigades  and  even  subordinate 
battalions  may  be  tasked  to  provide  peacekeeping  or  humanitarian  assistance.  In  preparing  for 
this  lesson,  interviews  were  conducted  with  bat-talion-level  oflBcers  who  were  responsible  for 
the  management  of  dislocated  civilian  during  Operation  Provide  Comfort.  As  a  result,  this 
lesson  was  written  as  an  introduction  to  establishing  a  dislocated  civilian  camp.  It  is  designed  as 
a  common  sense  approach  for  a  non-civil  affairs  unit  tasked  with  developing  a  camp. 

Question  #15  -  Why  waste  time  asking  questions  like  this? 

Response:  No  change.  The  question  concerns  animal  control,  particularly  in  a  Third  World 
environment.  This  is  a  critical  health  and  planning  consideration. 

Need  to  ask  questions  that  apply  to  real  life  scenarios. 

Response:  No  change.  Both  Lesson  2C  and  its  accompanying  practical  exercise  were  based  on 
a  given  scenario.  Operation  Provide  Comfort. 
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Give  the  user  the  alternative  to  select  screen  colors. 

Response:  No  change.  Screen  background  has  been  standardized. 

Check  dislocated  civilian  section. 

Response:  No  change.  Lesson  2C  was  reviewed  for  accuracy. 

COMPREHENSIVE  END  OF  EXAM  (COMPS) 

D3F98F  (3)  Don’t  know,  the  order  that  I  was  given  for  this  BETA  TEST  had  little  S5,  CA, 
PSYOP. 

Response:  No  change.  Question  does  not  refer  to  a  specific  OPORD,  nor  is  one  required  to 
answer  the  question. 

D304M  (5)  Scored  incorrect  but  feedback  say  “Tactical  Planning  Team”. 

Response:  No  change.  Correct  answer  is  indicated  and  scored  properly. 

D3E49E  (10)  Get  rid  of  this  question,  does  not  make  sense. 

Response:  Question  deleted. 

D3E49E  (10)  Not  an  S5  responsibility,  “S5  has  no  business  being  there”. 

Response:  Question  deleted. 
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APPENDIX  E 
Final  Research  Report  Data 


1.  Tliroughout  Beta  testing,  testers  recorded  the  time  they  spent  studying  the  various  course  components. 
The  reading  time  for  text  based  lessons  shown  in  the  following  chart  was  computed  based  on  the  estimated 
time  to  read  the  Student  Workbook  lessons  at  12  pages  per  hour.  Since  reading  time  will  vary  between 
students  depending  on  their  experience,  and  reading  abilities,  this  estimated  value  is  provided  to 
anticipate  the  amount  of  time  a  student  might  need  to  complete  a  course  of  instruction.  (Actual  student 
reading  time  is  addressed  in  a  subsequent  chart.)  The  CBI  study  time  was  computed  based  on  the  average 
actual  time  spent  by  Beta  testers  studying  the  CBI  material  and  completing  the  exams.  According  to 
student  reported  iirformation,  many  testers  did  not  read  any  doctrinal  material  for  the  lessons.  Based  on 
student  experience  and  proficiency  level,  the  amount  of  time  needed  to  study  doctrine  to  ensure  entry  level 
skills  for  a  course  will  vary  significantly.  This  chart  does  not  include  any  time  for  reading  doctrinal 
material. 


STUDY  TIME  (HOURS) 


COURSE 

TEXT 

CBI 

BRIGADE  COURSES 

Common  Core 

13 

18.1 

Commander 

34 

33.7 

Executive  Officer 

17 

10.2 

SI  Officer 

13 

9.4 

S2  Officer 

10 

13.5 

S3  Officer 

12 

13.2 

S4  Officer 

16 

12.2 

S5  Officer 

14 

6.2 

S3  Air  Officer 

8 

11.5 

Fire  Support  Officer 

20 

8.8 

Air  Defense  Officer 

8 

12.2 

Signal  Officer 

3 

4.4 

Chemical  Officer 

8 

8.3 

Asst.  Brigade  Engineer 

11 

11.2 

TOTAL 

187 

172.9 

BATTALION  COURSES 

Common  Core 

9 

7.6 

Commander 

26 

20.3 

Executive  Officer 

16 

10.1 

SI  Officer 

13 

9.6 

S2  Officer 

7 

13.7 

S3  Officer 

9 

6.4 

S4  Officer 

15 

7.2 

Chaplain 

5 

2.2 

S3  Air  Officer 

6 

10.9 

Fire  Support  Officer 

19 

6.3 

Air  Defense  Officer 

8 

5.1 

Signal  Officer 

4 

5.1 

Chemical  Officer 

9 

9.1 

Engineer  Officer 

8 

7.6 

TOTAL 

154 

121.2 
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2.  The  following  chart  shows  the  number  of  lessons  contained  in  the  ITTBBST-BSTS  com^s.  The 
column  for  “TEXT’  means  the  lesson  material  is  entirely  text-based,  except  for  the  lesson  exam.  The 
exam  for  every  lesson  in  every  course  is  contained  in  CBI.  The  “CBl”  column  means  the  lesson  is  entirely 
CBI-based,  unless  the  student  reads  doctrinal  reference  material.  The  colunm  labeled  “TEXT/CBI” 
means  part  of  the  lesson  is  text-based  and  part  of  the  lesson  is  CBI-based  (either  tutorial  or  practical 
exercise.) 


NUMBER  OF  LESSONS 


3.  Each  of  the  subjects  within  each  of  the  courses  contains  a  pretest  which  the  student  must  complete  in 
order  to  receive  credit  for  the  course  within  tlie  training  management  system  and  be  allowed  to  take  the 
lesson  exams.  A  post  test  is  provided  at  the  end  of  each  subject.  Scores  in  the  below  chart  were  computed 
by  averaging  all  subject  test  scores,  for  all  Beta  testers,  for  all  Subjects  within  the  designated  course.  The 
“improvement”  is  tlie  difference  between  tlie  pretest  and  post  test  scores. 


PRETEST/POST  TEST  IMPROVEMENT 


4.  Every  lesson  in  every  subject  of  every  course  contains  a  lesson  exam  that  is  CBI-based.  Scores  in  this 
chart  were  computed  by  averaging  all  lesson  exam  scores,  for  all  Beta  testers,  for  all  lessons  within  the 
designated  course. 


LESSON  EXAM  SCORES 


COURSE 

EXAM  SCORES 

BRIGADE  COURSES 

Common  Core 

93 

Commander 

92 

Executive  Officer 

85 

SI  Officer 

89 

S2  Officer 

85 

S3  Officer 

88 

S4  Officer 

88 

S5  Officer 

86 

S3  Air  Officer 

93 

Fire  Support  Officer 

92 

Air  Defense  Officer 

87 

Signal  Officer 

91 

Chemical  Officer 

87 

Asst.  Brigade  Engineer 

88 

AVERAGE 

89 

BATTAUON  COURSES 

Conunon  Core 

89 

Commander 

80 

Executive  Officer 

91 

SI  Officer 

92 

S2  Officer 

81 

S3  Officer 

89 

S4  Officer 

84 

Chaplain 

90 

S3  Air  Officer 

85 

Fire  Support  Officer 

86 

Air  Defense  Officer 

92 

Signal  Officer 

92 

Chemical  Officer 

93 

Engineer  Officer 

89 

87 


AVERAGE 


5.  A  CBI-based,  comprehensive  exam  is  provided  at  the  end  of  each  course.  Scores  in  this  chart  were 
computed  by  averaging  all  Comprehensive  exam  scores,  for  all  Beta  testers  within  the  designated  course. 

COMPREHENSIVE  EXAM  SCORES 


6.  Throughout  Beta  testing,  testers  recorded  the  time  they  spent  studying  the  various  course  components. 
The  estimated  text  hours  was  computed  based  on  the  time  to  read  the  Student  Workbook  text-based 
lessons  at  12  pages  per  hour.  The  actual  text  hours  was  computed  by  averaging  the  actual  time  spent 
reading  the  lessons,  as  recorded  by  Beta  testers.  Actual  reading  time  is  expected  to  vary  between  students 
based  on  experience  level  and  reading  abilities.  However,  some  students  did  not  read  selected  text-based 
lessons.  Tlie  actual  text  reading  time  is  about  one  third  of  the  estimated  text  reading  time  averaged  over 
all  courses.  The  estimated  doctrine  hours  was  computed  based  on  the  time  to  read  the  Required  Reading 
sources  listed  with  each  lesson  at  12  pages  per  hour.  Where  sections  of  doctrinal  references  were  listed  as 
Required  Reading  in  more  than  one  lesson,  tlie  reading  time  was  only  counted  once.  The  actual  doctrine 
reading  hours  was  computed  by  averaging  the  actual  time  spent  reading  the  doctrinal  references,  as 
recorded  by  Beta  testers.  Actual  doctrine  reading  time  is  expected  to  vary  between  students  based  on 
experience  level  and  reading  abilities.  However,  several  students  did  not  read  doctrinal  references  for 
most  of  the  lessons.  Averaged  over  all  Beta  testers  for  all  courses,  the  students  spent  only  about  3  percent 
of  the  time  estimated  for  reading  doctrinal  references.  This  equates  to  an  average  of  about  50  minutes  per 
student  per  course. 


REDING  TIMES  (hours) 


TEXT 

TEXT 

DOCTRINE 

DOCTRINE 

COURSE 

ESTIMATED 

ACTUAL 

ESTIMATED 

ACTUAL 

BRIGADE  COURSES 

Common  Core 

13 

3.1 

93 

1.5 

Commander 

34 

12.6 

170 

1.1 

Executive  OflScer 

17 

5.2 

75 

1.3 

SI  Officer 

13 

4.2 

53 

4.4 

S2  Officer 

10 

2 

91 

6.4 

S3  Officer 

12 

3.7 

103 

2.1 

S4  Officer 

16 

9.2 

44 

6.5 

S5  Officer 

14 

8.8 

43 

2.8 

S3  Air  Officer 

8 

3.1 

60 

2.6 

Fire  Support  Officer 

20 

3.5 

108 

2.8 

Air  Defense  Officer 

8 

4.1 

72 

8.3 

Signal  Officer 

3 

1.7 

35 

1.6 

Chemical  Officer 

8 

2.5 

38 

1.5 

11 

4.3 

121 

5.1 

BATTALION  COURSES 

Common  Core 

9 

2.1 

92 

0.4 

Commander 

26 

7.3 

163 

5.2 

Executive  Officer 

16 

5.3 

74 

0.1 

SI  Officer 

13 

8.9 

48 

0.7 

S2  Officer 

7 

2.3 

63 

0.2 

S3  Officer 

9 

2 

63 

0.2 

S4  Officer 

15 

5 

39 

3.3 

Chaplain 

5 

1.3 

33 

0 

S3  Air  Officer 

6 

2.7 

74 

0 

Fire  Support  Officer 

19 

2.3 

145 

0.8 

Air  Defense  Officer 

8 

1.7 

63 

2.1 

Signal  Officer 

4 

1.3 

44 

0.3 

Chemical  Officer 

9 

2.5 

27 

1.7 

Engineer  Officer 

8 

4 

106 

3.3 
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7 .  At  the  completion  of  each  Subject  for  each  course,  the  students  were  asked  to  rank  order  the  course 
components  as  to  their  educational/training  value.  The  columns  in  the  below  chart  represent  the  course 
components; 

-Workbook;  Student  Workbook  text  lessons. 

-Job  Aids;  Checklists  and  guides  contained  in  the  back  of  the  Student  Workbook,  that  can  be 
reproduced  and  used  by  staff  officers  to  assist  tliem  in  the  performance  of  their  duties. 

-References;  Doctrinal  publications  listed  as  Required  Reading  for  each  of  the  lessons. 

-CBI;  Computer-based  training  material  (tutorials  and  practical  exercises). 

-Exercises;  Subject,  lesson,  and  COMPS  exams. 


The  numbers  in  this  chart  were  computed  by  averaging  the  ratings  based  on  the  assessment  by  all  Beta 
testers  within  the  designated  course.  In  this  chart  “1”  means  the  most  valuable  and  “5”  means  the  least 
valuable 

STUDENT  ASSESSMENT  OF  COURSE  COMPONENTS 


WORKBOOK  I  JOB  AIDS 


REFERENCES 


COURSE 


BRIGADE  COURSES 


Common  Core 


Commander 


Executive  Officer 


SI  Officer 


S2  Officer 


S3  Officer 


S4  Officer 


S5  Officer 


S3  Air  Officer 


Fire  Support  Officer 


Air  Defense  Officer 


Signal  Officer 


Chemical  Officer 


BATTALION  COURSES 


Common  Core 


Commander 


Executive  Officer 


SI  Officer 


S2  Officer 


S3  Officer 


S4  Officer 


S3  Air  Officer 


Fire  Support  Officer 


Air  Defense  Officer 


Signal  Officer 


Chemical  Officer 


Engineer  Officer 


TOTAL  (Lowest  is  best) 


5 

1 

2 

I 

5 

3 

2 

4 

1 

2 

5 

1 

2 

3 

5 

1 

4 

2 

1 

5 

2 

2 

5 

3 

27 

59 

45 

EXERCISES 


TOTAL  (Lowest  is  best) 

29 

54 

46 

26 

40 

2 

3 

1 

3 

2 

3 
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APPENDIX  F 


Review  of  Related  Research 

Previous  Advanced  Research  Projects  Agency  (ARP A)  and  ARI  projects  have  directly 
attacked  the  battlestaff  competency  issue.  The  first  project,  by  the  Infantry  Forces  Research  Unit, 
produced  the  Commander’s  Battle  Staff  Handbook  Thompson,  &  Valentine,  1993) 

which  serves  as  an  interim  solution  to  staff  individual  training  and  synchronization  issues.  It  is  a 
description  of  individual  staff  position  duties,  responsibilities,  and  references  that  is  being 
distributed  to  the  Reserve  Components  (RC)  and  Active  Components  (AC). 

The  second  project  was  ARPA  sponsored  under  the  program  umbrella  of  Simulation  In 
Training  for  Advanced  Readiness  (SIMITAR).  Applying  the  Commander's  Battle  Staff 
Handbook  as  the  start  point  of  the  TD  process,  BN-BSTS  for  Army  National  GKiard  (ARNG) 
Units  was  developed  and  delivered  to  the  ARI  in  January  1995.  Likewise,  applying  the  Brigade 
Commander’s  Battle  Staff  Handbook  (fenAxQ,  &  Valentine,  1996)  led  to  the  development  of 
BDE-BSTS,  delivered  to  ARI  in  January  1996.  These  programs  focused  on  the  development  of 
prototype  battlestaff  officer  training  programs  to  be  available  to  ARNG  officers  at  home  or  in  the 
armory.  These  multimedia  prototype  training  programs  have  been  distributed  to  the  two  ARNG 
test  brigades  selected  for  training  at  the  NTC  in  1996-1997.  Portions  are  also  in  use  by  the  US 
Army  Infantry  School,  the  US  Army  Armor  School,  and  the  US  Army  Aviation  Center.  The  CSS 
Battle  Staff  Training  System  (CSS-BSTS)  developed  for  the  ARNG  used  the  approach  designed 
for  BN  and  BDE-BSTS  to  develop  required  battlestaff  training  for  the  Forward  Support  Battalion 
(FSB)  of  the  divisional  and  separate  brigade. 

The  applicable  successes  and  lessons  learned  from  the  BN-BSTS,  BDE-BSTS,  CSS- 
BSTS  as  well  as  the  parent  program,  SIMITAR,  and  the  family  of  collective  training  programs 
(SIMBART,  SIMUTA,  and  COBRAS)  were  alt  studied  in  the  course  of  the  ITTBBST-BSTS 
project.  The  Center  for  Army  Lessons  Learned  (CALL)  publications  were  essential,  since  they 
contained  lessons  learned  from  reports,  data,  and  observations,  both  from  combat  operations  and 
from  CTC  After  Action  Reviews  (AAR).  Additionally,  the  wealth  of  knowledge  and  information 
available  from  the  CTC  Operations  Groups,  at  both  the  NTC  and  JRTC,  as  well  as  the  subject 
matter  experts  (SMEs)  at  Ft.  Leavenworth’s  Battle  Command  Training  Directorate  (BCTD)  and 
Battle  Command  Battle  Lab  (BCBL)  were  tapped  to  enhance  the  existing  BSTS  training 
programs  for  ITTBBST-BSTS.  These  information  sources,  in  conjunction  with  existing  doctrinal 
publications  and  Tactics,  Techniques,  and  Procedures  (TTP)  were  integrated  into  the  TSPs  for 
battalion  and  brigade  commanders  and  their  staff  officers. 
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APPENDIX  G 


Hardware  and  Software  to  Operate  BSTS 


HARDWARE  (installed  and  configured): 


486  DX2  Computer,  with  66  MHZ  processor 

8  Mb  RAM,  expandable  to  32+Mb;  70ns  or  better 

200  Mb  Hard  Disk,  or  better 

16  bit  local  bus 

101  Key  Enhanced  Keyboard 

Desktop  cabinet 

Serial  Mouse  with  driver 

Sound  Blaster  16  Sound  board,  or  compatible,  with  driver  and  speakers 
105K/300K  double  spin  CD-ROM  or  better 
3-1/2”,  1.44  Mb  disk  drive 

Spider  Graphics  accelerator  card,  or  equivalent,  with  driver  to  handle  640x480, 
256  color  and  14-17”  monitor,  1MB  RAM  or  better 
14-15”  monitor,  .28  dpi 


EITHER  THE  BELOW  LISTED  MODEM  (WAN)  OR  ETHERNET  CARD 

(LAN) 


(not  required  for  stand-alone  mode.) 


9600  bps,  or  better,  modem  with  lOBT  RJ45  connector 

Ethernet  card  with  connector  to  meet  LAN  configuration  requirements 

SOFTWARE  (installed  and  configured): 

DOS  6.21 

Windows  for  Workgroups  3.1 1  w/Video  for  Windows 
Icon  Author  Present  7.0 
EMMii  student  software 

ACCESSORIES 


Surge  Strip 

10BT/RJ45  modular  plug  with  10’  cable  (WAN) 
Mouse  Pad 
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